Personalized interactive virtual environment in vehicle

Abstract

A method for providing an interactive virtual communication environment within a vehicle uses real-time sensor data. The method obtains data from one or more vehicle sensors, and then processes the data to obtain occupant-specific data. Using the occupant data, the method configures a virtual environment displayed on a display device of the vehicle. Within the virtual environment, an avatar representing a vehicle occupant is generated based on sensor data and associated occupant information. The avatar acts as an indicator of the presence and identity of the occupant within the virtual environment, providing a more accurate and real-time indication of whether the occupant can interact, thereby reducing the hardware complexity of the communication platform inside the vehicle.

Description

Technical Field

[0001] Various examples of the present disclosure generally relate to the field of interactive virtual environments in vehicles. Various examples of the present disclosure specifically relate to methods and systems for enabling personalized, virtual interactions for occupants within a vehicle using real-time sensor data and pre-configured user preferences. Background Art

[0002] The interaction of vehicle occupants with communication platforms presents a range of technical challenges. Traditional in-vehicle entertainment and communication systems are often limited to manual user interfaces for configuration and lack the level of interactivity and personalized communications that users expect. The user interface should be intuitive so that occupants can easily understand the virtual environment, while also being flexible enough to allow for a personalized experience. This task is complicated by the limited attention span of vehicle occupants, and the safety limitations that may arise from participating in public transportation.

[0003] Maintaining the user-friendliness of in-vehicle communication platforms is difficult due to the constantly changing conditions inside and outside the vehicle. Variables such as the number of occupants, speed, lighting, and temperature can fluctuate rapidly, which often makes it challenging to maintain a stable and user-friendly virtual experience. Safely interacting with communication platforms while driving a vehicle presents challenges.

[0004] Traditional methods of indicating presence or status, such as changing an online profile or sending a message, require visual focus and manual manipulation of the device. Constantly changing conditions inside and outside the vehicle can limit the user's ability to overview the availability of others within the virtual environment, causing discomfort and potentially resulting in poor communication. Summary of the invention

[0005]

[0006] Therefore, there is a need for advanced techniques for providing a communication platform, and in particular a virtual communication environment in a vehicle, that alleviates or mitigates at least some of the limitations and disadvantages identified above.

[0006] This need is met by the features of the independent claim. Features of the dependent claims define further advantageous examples.

[0007] In the following, the solution according to the present disclosure will be described with respect to the claimed method and with respect to the claimed computing device, virtual environment system and vehicle, wherein features, advantages or alternative embodiments may be assigned to other claimed objects and vice versa. In other words, the claims related to the device and the system may be improved with features described in the context of the method, and the method may be improved with features described in the context of the device and the system.

[0008] A method for providing an interactive virtual environment, in particular a virtual communication environment, within a vehicle is provided. In particular, the interactive virtual environment may be provided to an occupant while using the vehicle, in particular to a driver while driving the vehicle. The virtual environment may enable communication with other remote persons who are not in the vehicle and are represented in the virtual environment.

[0009] In one step, sensor data from at least one vehicle sensor is acquired. The interior and / or exterior of the vehicle is sensed to collect data associated with or corresponding to an occupant, which can be used to identify the presence and / or identity of the occupant within the vehicle. The sensor data can be received by a processing device (in other words, a computing device) of the vehicle, which further processes the sensor data.

[0010] In various examples, sensor data is obtained from at least one vehicle-mounted sensor that can monitor the interior (e.g., a cabin) or exterior (e.g., traffic or surroundings of the vehicle) of the vehicle. Among the various types of sensors, the at least one vehicle sensor can include, for example, one of an in-cabin camera, a fatigue detection sensor, a seat-embedded pressure sensor, or a digital key sensor.

[0011] At least one sensor may face both the interior and exterior of the vehicle and / or be located in both the interior or exterior of the vehicle. Vehicle sensors may be attached to the vehicle. Interior sensors may include an in-cabin camera located on the dashboard, seat embedded pressure sensors in the driver and passenger seats, and fatigue detection sensors integrated into the steering wheel. External sensors may include door handle sensors or digital key sensors that detect proximity, or cameras that sense the external environment of the vehicle. These various sensors are used to identify people, such as occupants of the vehicle, and further, for example, to distinguish between the driver and other occupants.

[0012] For example, an in-cabin camera can capture facial features and gestures and transmit that data over the vehicle's internal communications network, such as a controller area network (CAN) bus. Seat-embedded pressure sensors can detect the occupant's weight and posture and similarly input that data. Door handle sensors or digital key sensors can detect the occupant's initial approach and entry, distinguishing between the driver and the passenger.

[0013] Sensor data acquired from multiple interior and exterior sensors may be fused and processed simultaneously to enhance detection and identification of occupant presence. For example, an in-cabin camera may capture facial features, while seat-embedded pressure sensors may provide data on weight and posture. Fatigue detection sensors on the steering wheel may provide additional behavioral data, such as grip patterns. All of this sensor data may be processed individually or fused to be included in the sensor data, which may be processed to determine occupant data indicative of the occupant's presence and / or identity.

[0014] In one example, the processing device may employ a machine learning algorithm or decision tree to integrate sensor data (including, for example, facial recognition data, weight, posture, and grip patterns) to accurately identify the occupant. In other words, rather than relying on data from a single sensor, the system fuses multiple sensors to create a more reliable and robust presence detection and identification mechanism. This multi-sensor approach is particularly beneficial in situations where the data from one of the sensors may be obscured or corrupted, as the system can rely on data from other sensors to ensure accurate presence detection and identification.

[0015] Therefore, in a further step, the sensor data is processed to determine occupant data associated with an occupant inside the vehicle. In other words, occupant data is generated or calculated based on or using the sensor data. The processing may be performed by a processing device or computing device in the vehicle.

[0016] In various examples, processing the sensor data may include detecting an occupant entering the vehicle and identifying the occupant. In various examples, the processing device may determine whether the occupant is a driver or a passenger, and whether and / or where they are located in the vehicle cabin. In other words, based on and / or using the sensor data, occupant data is determined, which may include information indicating the entry and / or presence of the occupant, as well as identification of the occupant.

[0017] The sensor data may be processed by an onboard processing device to determine occupant data associated with an occupant within the vehicle. This may involve detecting an occupant entering the vehicle and identifying the occupant. In various examples, the processing device may determine whether the occupant is a driver or a passenger, and whether they are in the vehicle cabin.

[0018] In various examples, the sensor data is processed to determine occupant data, the occupant data including an identification of an occupant within the vehicle. Specifically, this may involve using facial recognition or biometric sensors to verify the identity of the occupant. In various examples, the processing device may determine whether the occupant is a driver or a passenger based on the sensor data. The identity may be used to load personalized settings or preferences in the interactive virtual environment.

[0019] In various examples, based on and / or using sensor data, the physical state of the occupant can be estimated or determined and represented in the virtual environment. In addition to presence detection and identification, sensor data acquired from various internal and external vehicle sensors can also be processed to determine the physical state or health of the occupant. For example, temperature sensors inside the cabin can detect whether the occupant's body temperature is elevated, indicating discomfort. Data from pressure sensors in the seats can analyze the occupant's posture and movement, potentially detecting discomfort or anxiety. Sensors embedded in seat belts or radar sensors can track breathing rate. In-cabin cameras with computer vision capabilities can potentially analyze facial expressions and body language to assess emotions, pain levels, or alertness.

[0020] Many modern vehicles already contain an array of sensors that monitor vehicle status for other purposes such as vehicle control, entertainment, and safety. With proper processing, data from these existing sensors can also provide insights into the physical condition of the occupants. For example, pressure sensors on seats used to detect occupancy to deploy airbags can also analyze fidgeting or frequent movement that could indicate discomfort. Microphones used for hands-free phone calls can potentially process vocal tones and patterns to detect signs of illness, frustration, or fatigue. Accelerometers and motion sensors used for vehicle stability control can also register erratic hand gestures or driving behaviors that could be associated with an occupant’s health or impaired abilities.

[0021] By processing various sensor data sources already embedded in modern vehicles, the system can make a computer-aided determination of the occupant's physical condition, health status, or well-being. In addition to presence detection and identification, this can also generate a more accurate digital profile of the occupant, which can be relevant for display in a virtual environment when talking to another person in the virtual environment. The occupant's physical state data can then be incorporated and represented in the virtual environment, for example by changing the appearance, posture, and / or movement of the corresponding avatar during interaction and conversation.

[0022] In various examples, several types of processing devices can perform the calculations required for occupant identification based on sensor data. A central processing unit (CPU), often referred to as a vehicle control unit, acts as the main computing center for coordinating various functions and processing data. This unit can be responsible for integrating sensor data and applying algorithms for occupant identification.

[0023] In various examples, domain controllers such as infotainment systems may also perform specialized processing tasks in addition to the CPU. For example, an infotainment domain controller may handle processing related to occupant entertainment preferences, or even certain aspects of authentication if linked to a personal mobile device or user account.

[0024] Regional architecture computing devices are another important component in this context. These are specialized computing units designed to process sensor data and controls specific to a particular zone within the vehicle, such as the driver's area or the back seat. For example, the computing unit can process data from seat-embedded pressure sensors or in-cabin cameras located in its designated zone to facilitate the overall occupant identification process.

[0025] Other examples of processing devices may include dedicated units for advanced driver assistance systems (ADAS), which also collect and process sensor data for various safety features. These ADAS systems may share relevant data (such as fatigue detection measures) with the main CPU or domain controller for more comprehensive occupant identification.

[0026] These various computing devices typically communicate over the vehicle's internal network, such as a controller area network (CAN) bus, to share and coordinate sensor data for processing. By distributing the computational load to specialized units, modern vehicles can perform complex tasks such as occupant identification more efficiently and reliably.

[0027] In a further step, the interactive virtual environment is configured by the processing device based on and / or using the determined occupant data.Alternatively, the virtual environment may be first started or initially displayed to the occupant.

[0028] The interactive virtual environment may be configured based on the determined occupant data and thereby based on or using the vehicle sensor data. The configuration may include a meeting area and a 3D model of the occupant. In other words, the environment may be arranged to include a virtual representation of both the occupant and an external meeting area, the external meeting area representing a real meeting area where the occupant and other persons would normally communicate face-to-face, i.e. in real life.

[0029] In various examples, a processing device within the vehicle can use the occupant data to perform steps to launch and / or configure the virtual environment. It can also transmit at least a portion of the occupant data over a communication network to receive additional data to further refine the virtual environment. For example, when the occupant is identified as a driver, the device can send a query to a server to retrieve the driver's favorite playlist or news channel to incorporate into the virtual space.

[0030] Alternatively, at least part of the configuration of the virtual environment can be performed externally based on the transmitted occupant data. In this case, the processing device transmits the occupant data to an external server, which then performs the configuration. The virtual environment data is then sent back to the vehicle to start the virtual environment. The data may include information or 3D models about different persons outside the vehicle, which persons can be represented in the virtual environment.

[0031] In another step, the interactive virtual environment is displayed to the occupant. The display may be performed on a vehicle display device. Various types of display devices may be used, including an integrated infotainment screen or a head-up display projected onto the windshield.

[0032] Interactive virtual environments can be presented to vehicle occupants on a variety of display devices integrated into the vehicle. Various visual, audio, and tactile display technologies within the vehicle cabin can be used to project interactive virtual experiences to occupants to create an immersive user environment.

[0033] In various examples, the main infotainment screen is typically located in the center console area. These high-resolution LCD / OLED displays are driven by a graphics processing module to render the virtual environment. The central infotainment screen is visible to both front and rear occupants. Another possibility is to use a head-up display (HUD) system that uses vehicle projection optics to overlay virtual interface graphics onto the windshield, allowing better communication within the virtual environment. HUD systems typically include a small projection unit and combiner optics mounted below the dashboard. The digital instrument panel behind the steering wheel can also serve as a display surface for virtual objects dedicated to the driver. Here, rendered elements can be overlaid onto the digital speedometer, tachometer, and other instrument graphics provided by the instrument panel. For passengers in the rear compartment, a dedicated rear seat entertainment display can be utilized to present the virtual environment. These displays are typically integrated into the back of the front seat headrests. Personal mobile devices brought into the vehicle, such as tablets or smartphones, can participate in the virtual environment by connecting them to the system's docking mechanism. Their onboard displays and speakers become an extension of the rendered environment.

[0034] Thus, through the use of vehicle sensors in combination with a virtual environment according to the techniques of the present disclosure, the complexity of interacting with the vehicle's communication system is reduced, particularly in the context of communicating with other people in a virtual environment. For example, sensors can automatically identify users and set preferences, thereby eliminating the need for manual setup each time the vehicle is used. This simplification is particularly beneficial in a virtual environment, where ease of interaction is critical to an immersive experience, making it easy to present the presence and identity of all communicating persons.

[0035] The technology facilitates seamless integration of external elements (such as 3D models or real-time data) into the virtual environment by exchanging information between the vehicle communication bus and personal communication channels (such as a communication platform and a virtual environment with an avatar). This eliminates the need for the occupant to manually input or update such information, thereby reducing interaction complexity.

[0036] The sensor can be selected from a group of sensors including but not limited to in-vehicle cameras, fatigue detection sensors, seat-embedded pressure sensors, smartphone-enabled NFC or Bluetooth sensors, digital key sensors, and in-cabin radar sensors.

[0037] In-cabin cameras can be positioned to capture the interior of the vehicle, specifically the driver and passenger seating and activity areas. This type of sensor can acquire real-time video or image data of the vehicle occupants.

[0038] Fatigue detection sensors can use infrared technology to monitor the driver's eye movements and facial expressions. Fatigue detection sensors use infrared emitters to illuminate the driver's face with invisible light. The reflected infrared light is captured by an infrared camera sensor. This light data is processed to analyze eyelid movement, gaze, head posture, etc. to detect signs of fatigue. The sensor can provide data representing eye closure rate, blink duration, and facial expressions.

[0039] Seat embedded pressure sensors can be integrated into the seat cushions and backrests of a vehicle. These sensors can detect the occupant's weight, pressure distribution, and posture, which can then be used to adjust the seat or climate control settings for optimal comfort. Seat pressure sensors use an array of force-sensitive resistors embedded in the seat cushions and backrests. These resistors change resistance based on the pressure applied. The sensors detect the distribution of weight and force on the seat surface. The data can represent the occupant's weight, posture, position, and movement.

[0040] Smartphone-enabled NFC (Near Field Communication) or Bluetooth sensors allow secure communication between the vehicle's systems and the smartphone. By doing so, it can enable features such as personalization, authentication, or even remote control of certain vehicle functions. Smartphone-enabled sensors use NFC or Bluetooth protocols for proximity-based communication with paired smartphone devices. They transmit radio signals that can identify the phone and extract basic identification or authentication data from it. This can provide data indicating the identity and presence of the occupant carrying the smartphone.

[0041] The Digital Key Sensor acts as an electronic lock and enables vehicle entry without the need for a traditional mechanical key. The sensor can recognize a secure digital token from a dedicated device or even a smartphone app. The Digital Key Sensor is an RFID reader that reads the unique ID from the RFID tag in the Digital Key device. It provides presence and authentication data to identify the occupant carrying the Digital Key and authorize vehicle access.

[0042] The In-Cabin Radar Sensor uses radio waves to detect and monitor the movement and position of vehicle occupants. It can be particularly useful for assessing the vehicle interior in low-light or obstructed conditions, where camera sensors may be less effective. The In-Cabin Radar Sensor transmits low-power radio waves and uses radar technology to analyze the reflected waves. It detects the occupant's position, movement, posture, breathing, etc. It provides positional data that locates the occupant in 3D space within the vehicle cabin.

[0043] Sensor data from various types of sensors can be combined or fused to gain a comprehensive understanding of the occupant and thus configure the virtual environment.

[0044] Processing the sensor data may include detecting entry and / or presence of an occupant in the vehicle, and the occupant data may include an indication of the presence of an occupant in a cabin of the vehicle.

[0045] In various examples, a camera can analyze image frames to detect the motion of an entering occupant. Background modeling can distinguish between empty and occupied seats. Seat pressure sensors can detect changes in weight and pressure distribution, indicating seat occupancy. The mere detection of a paired smartphone by Bluetooth or NFC indicates an occupant has entered. Authentication of a digital key sensor also implies the presence of the key owner. Radar sensors also provide motion and seat area occupancy data. By processing sensor data including one or more of the above input data, a processor can determine an occupant entry event and the presence of an occupant inside the vehicle.

[0046] Processing the sensor data may include determining an identity of an occupant, and the occupant data may include identification data representative of the identity of the occupant.

[0047] In various examples, smartphone sensors and digital key sensors can provide identity information. A Bluetooth MAC address or NFC ID can identify a smartphone and its user. A digital key ID corresponds to the assigned key owner. Cameras and radars can also apply biometric analysis at entry. Facial recognition algorithms can identify occupants through camera images. Feature data from radar sensors can be used for identity determination. The system can maintain user profiles associated with identified occupants for personalization. In addition, location sensors (such as GPS sensors) can be used to provide sensor data. Acoustic sensors inside and outside the vehicle can also be used as vehicle sensors.

[0048] The system can process raw sensor data to detect the presence and / or identify occupants through various techniques. For example, algorithms can analyze unique physiological features such as fingerprints, vein patterns, or facial geometry to identify occupants by matching them to profiles in a database. The system can maintain a lookup table that associates unique digital IDs from devices or keys with corresponding user profiles. Sensor data containing these digital IDs allows identity lookup. Machine learning models can be trained to recognize different patterns in sensor readings that are unique to individuals. Fingerprints or signatures for specific identities can be retrieved from a database. New sensor data is matched to these fingerprints for identification. Rules can be defined that compare sensor values ​​to preset thresholds or ranges that are unique to each profile. Readings within the expected range are meant to match a certain occupant. Fusion algorithms can combine sensor readings from multiple inputs and calculate similarity scores with profiles. The highest similarity score determines the matching profile and therefore the occupant's identity. Over time, the system can build dynamic profiles of individuals by associating sensor data with feedback such as user input. Sensor patterns that identify recurring occupants are allowed to emerge. Occupant data may include a tag, name, or unique ID that ties the sensor reading to a matching user profile representing the identified occupant.

[0049] Thus, vehicle sensor data processing allows for robust entry, presence and identity detection represented by occupant data. The processor may then use the sensor data and / or occupant data to configure or customize the virtual environment accordingly.

[0050] Configuring the virtual environment may include obtaining a 3D model representing a meeting area exterior to the vehicle and a 3D model representing the occupant based on the occupant data, and configuring an interactive virtual environment depicting the 3D model of the meeting area and the occupant.

[0051] In various examples, the processing device receives a detailed 3D model of an external meeting area from a memory or through a communication network in response to the occupant data. The meeting area model represents an actual physical location, such as a real location where the occupant typically interacts or communicates with others outside the vehicle in a private space. For example, the meeting area is a room in a house where the occupant will meet with others in real life after arriving at the destination and after exiting the vehicle.

[0052] A 3D model (avatar) of the occupant can be generated, which represents the occupant's body shape, clothing and posture. The 3D occupant model is then embedded and positioned in a virtual position within the rendered 3D meeting area model. Real-time rendering and adjustment of the occupant's 3D model based on / using one or more vehicle sensors can create an interactive virtual environment that depicts the communication space where the meeting will occur, with the 3D occupant model digitally located within the space. This allows the occupant to experience an immersive preview of the meeting location and imagine themselves participating in the upcoming communication and interaction. The virtual environment provides a digital replica of the physical meeting area, integrating the occupant model into the location before the occupant actually arrives at the location so that others can also feel that the occupant is in the meeting area they know.

[0053] The method may also include notifying another person outside the vehicle with whom the occupant intends to communicate of the occupant's presence in the virtual environment.

[0054] In various examples, the method may include notifying another person depicted in the virtual environment, to whom the virtual environment with the occupant's avatar is displayed and with whom the occupant intends to communicate, of the occupant's presence in the virtual environment. This allows external contacts of the occupant (such as family or friends) to join the occupant's interactive virtual environment, and these contacts may then also be depicted in the virtual meeting area, as will be described below.

[0055] The notification of the external contact can be performed in a variety of ways. In one example, the processing device retrieves a preferred communication channel for the external contact based on the occupant data. The preferred communication channel can be a phone number, email, social media account, or other communication channel configured by the occupant. The processing device sends a notification message to the external contact via the preferred communication channel indicating the occupant's presence in the virtual environment.

[0056] The method may also include obtaining a 3D model representing the other person, and configuring the interactive virtual environment using the 3D model of the other person.

[0057] In various examples, the method may also include obtaining a 3D model (avatar) representing another person outside the vehicle and using the obtained 3D model of the other person to depict the avatar at a virtual location in the interactive virtual environment. This allows a realistic representation of the external contact to be rendered in a meeting area of ​​the virtual environment along with the avatar of the vehicle occupant.

[0058] The 3D model (avatar) of another person may be obtained in a variety of ways. In one example, the processing device receives the avatar from a database or data storage device in the vehicle, in other examples, the avatar is transmitted via an external communication network in response to the occupant data, i.e., the processing device receives a 3D model representing another person from a remote computing or cloud device via a communication network. The 3D model may have been generated by 3D scanning technology and stored in association with the user profile.

[0059] Thus, the interactive virtual environment displayed to the occupants may include avatars that visually represent the occupants and another person outside the vehicle. When the virtual environment is displayed to the participants, they can easily identify who in the virtual environment can communicate, and the corresponding preferred communication channel is automatically used for each person in the background.

[0060] In various examples, different communication channels may be used for notifications and interactions between occupants and external contacts represented as avatars in the virtual environment. Audio channels, where the virtual environment can enable voice calls or video calls with audio between occupants and contacts using speakers, microphones, and audio processing in the vehicle. Text chat, where text messages can be exchanged in real time between occupants and contacts in the virtual environment using an in-vehicle display and input devices such as a touch screen. Augmented reality, where an avatar of a contact can be visually depicted in an interactive virtual environment and animated to speak and gesture when audio or text is received from the contact. Pre-recorded audio / video, where an audio or video recording made by a contact can be played back to the occupant in the virtual environment when triggered by an avatar interaction. Synthesized speech or text-to-speech, where text received from a contact can be converted to realistic speech and played through speakers using text-to-speech technology. Speech synthesis, where voice input made by an occupant can be converted to text using speech recognition and transmitted to the contact as a text message. Gesture / gaze input, where an occupant's gestures or gaze patterns can control hands-free interaction with a contact's avatar using an in-cabin camera and computer vision. Haptics, where contacts can transmit coded vibration patterns to the steering wheel or seats to get the occupant's attention or convey an alert.

[0061] An appropriate communication channel may be automatically selected for the occupant and / or other persons in the virtual environment. The communication channel used to communicate with other persons may be determined based on sensor data and / or occupant data. The communication channel may be dynamically adjusted during a conversation based on or using vehicle sensor data.

[0062] Specifically, a 3D model representing an occupant may be processed by a computing device to create an avatar of the occupant that represents the occupant and conditions as sensed by the vehicle sensors. This may involve extracting key features from the 3D model, such as face, body shape, clothing, etc., and converting them into an avatar form suitable for display in a virtual environment. Similarly, a 3D model representing another person is processed to generate an avatar representing their likeness. The avatars of the occupant and the other person may then interact with each other and with the virtual environment, providing an immersive visual experience. The avatar of the occupant may be dynamically adjusted based on or using sensor data and / or the occupant's physical state and / or attention score.

[0063] These avatars may be animated and / or depicted as interacting with each other based on vehicle sensor data and / or occupant data.

[0064] In one example, the occupant interacting with the virtual environment is the driver of the vehicle. Another person outside the vehicle is a family member or friend of the driver. The meeting area represented within the virtual environment is an interior living area belonging to the family member or friend. For example, their public or shared living room. So in this example, when the driver is in the vehicle, they enter a virtual representation of their family or friend's living room environment. The driver's avatar and the avatars of the family members / friends generated from their 3D models can interact there. This provides the driver with a virtual tour experience through the interactive virtual environment, as if the driver were already in the real meeting area.

[0065] The method may also include retrieving a preferred communication channel of another person outside the vehicle. The preferred channel is used to enable interaction between the other person and the occupant within the virtual environment generated in the vehicle.

[0066] In one example, during startup / configuration of the virtual environment, an occupant may specify in the vehicle system or retrieved through a communication network that Facebook Messenger is a preferred channel for contacting their friends. Subsequently, when the occupant enters the vehicle, the system retrieves the preferred Facebook Messenger channel for the friend based on the occupant data. The system uses the channel to send a notification to the friend that the occupant is present in the virtual environment and invites the friend to join. Once the friend accepts the invitation, the system establishes a connection through the preferred Facebook Messenger channel between the friend and the occupant. This allows the friend and occupant to communicate and interact within the virtual environment through text, voice, video, or other means used by the preferred communication channel. The virtual environment is configured to facilitate this communication by rendering a representation of the communicating occupant in the virtual environment.

[0067] By retrieving and utilizing another person's preferred communication channel, a seamless and natural interaction experience may be provided to connect vehicle occupants with external contacts through a virtual environment.

[0068] The method may also include: obtaining additional sensor data from at least one vehicle sensor that monitors external traffic conditions around the vehicle; processing the sensor data and the additional sensor data to determine an attention score that quantifies the degree of attention required by the occupant to respond to the external traffic conditions; and configuring the interactive virtual environment to intuitively represent the attention score.

[0069] In one example, the method involves receiving or measuring additional sensor data from vehicle sensors that monitor external traffic conditions around the vehicle. For example, vehicle dynamic sensors such as accelerometers, wheel speeds, and other motion data can detect aggressive or unsafe driving behavior. This can also include cameras, radars, lidars, etc. that detect other vehicles, obstacles, lane markings, etc. This external traffic data is processed together with the occupant sensor data to determine an attention score that quantifies the degree to which the driver / occupant needs to pay attention to the road based on the current situation. For example, smooth traffic may have a lower score, while traffic congestion or dangerous conditions will have a higher score. The interactive virtual environment is then configured to intuitively represent the attention score to the occupant. Some examples of visual representations include: displaying an attention score value or attention level indicator as a plurality of color indicators within the virtual environment; changing the visualization of the avatar in the virtual environment based on the attention score, such as fading / transparent, reducing the size, or blurring the virtual scene when the attention score is high; superimposing a graphical alert / notification when the attention score exceeds a threshold; or adjusting the sound or music in the virtual environment based on the attention score.

[0070] Determination of the attention score allows the virtual environment system to dynamically adjust the virtual environment based on the external driving environment, particularly in relation to the occupant's avatar, to indicate the occupant's required attention to traffic, respectively, to indicate the occupant's attention that can be provided for a conversation. The visual representation can also provide instructions to the occupant to focus on driving when needed.

[0071] Corresponding in-vehicle or vehicle computing devices and in-vehicle virtual reality systems are provided, which may be configured to perform any method or combination of methods as described in the present disclosure.

[0072] The computing device and virtual reality system include a processing unit and a memory, wherein the memory includes instructions that, when executed by the processor, cause the computing device to perform the following steps. In one step, sensor data is received from at least one sensor on the vehicle. In one step, the received sensor data is processed to determine occupant data related to an occupant inside the vehicle. In one step, an interactive virtual environment is configured based on the determined occupant data. In one step, the configured interactive virtual environment is output for display to the occupant on a vehicle display device.

[0073] The vehicle includes a computing device or a virtual reality system as described in the present disclosure. For such a processing device, system and vehicle, similar technical effects as described in the present disclosure can be achieved.

[0074] It should be understood that the features mentioned above and those yet to be explained below may be used not only in the respective combinations indicated, but also in other combinations or independently, without departing from the scope of the present disclosure. Specifically, the features mentioned above and those yet to be explained below may be used not only in the respective combinations indicated, but also in other combinations or independently, without departing from the scope of the present disclosure.

[0075] Therefore, the above summary is intended only to give a brief overview of some features of some embodiments and implementations, and should not be construed as limiting.Other embodiments may include other features besides those explained above. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] These and other objects of the present invention will be appreciated and understood by those skilled in the art from the detailed description of the preferred embodiments and the following drawings, in which like reference numerals refer to like elements.

[0077] Figure 1 Schematically illustrated is an overview of a vehicle according to various embodiments.

[0078] Figure 2 A virtual environment in a vehicle according to various embodiments is schematically illustrated.

[0079] Figure 3 The steps of a method for providing an interactive virtual environment within a vehicle according to various embodiments are schematically illustrated.

[0080] Figure 4 A computing device configured to execute a method for providing an interactive virtual environment within a vehicle according to various embodiments is schematically illustrated. DETAILED DESCRIPTION

[0081] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be understood that the following description of the embodiments should not be understood in a limiting sense. The scope of the present invention is not intended to be limited by the embodiments described below or by the accompanying drawings, which are considered as illustrative examples of general inventive concepts. The features of the various embodiments may be combined with each other unless otherwise specifically noted.

[0082] The drawings should be considered as schematic representations, and the elements shown in the drawings are not necessarily shown to scale. Instead, the various elements are represented so that their functions and general purposes are apparent to those skilled in the art. Any connection or coupling between the functional blocks, devices, components or other physical or functional units shown in the drawings or described herein may also be achieved by indirect connections or couplings. Coupling between components may also be established by wired connections. Functional blocks may be implemented in hardware, firmware, software or a combination thereof.

[0083]

[0013] Techniques related to providing a virtual communication environment for vehicle occupants, particularly while using and / or driving a vehicle, are described below.

[0084] Figure 1 Schematically an overview of a vehicle 1 according to various embodiments is shown.

[0085] The vehicle 1 includes interior or in-cabin vehicle sensors 2. These sensors 2 acquire signals from the interior of the vehicle and provide sensor data that can be processed to determine various types of occupant data, such as the presence and identity of the occupants. In various examples, the sensors may include in-cabin cameras, fatigue detection sensors, seat-embedded pressure sensors, smartphone-enabled NFC or Bluetooth sensors, digital key sensors, and in-cabin radar sensors. In addition, external sensors 3 that monitor the exterior of the vehicle can acquire data on external traffic conditions. The vehicle 1 also includes a display device 4, which may be an integrated infotainment screen, a head-up display projected onto the windshield, a digital instrument panel, a rear-seat entertainment screen, or a detachable tablet or mobile device docked in the vehicle.

[0086] Figure 2 A virtual environment 5 in a vehicle according to various embodiments is schematically illustrated.

[0087] The virtual environment 5 includes an avatar 6, which represents an occupant inside the vehicle. Avatar 6 is generated based on occupant data from sensors 2 inside the vehicle cabin. In this way, the avatar serves as an indication that the occupant is present and ready to communicate. An additional avatar 7 represents a woman, who is now outside the vehicle. The presence of avatar 7 in the virtual environment serves as an indication that the woman is also ready to communicate. Both avatars 6 and 7 are located in a meeting area 8, which is a 3D model representation of a physical location outside the vehicle, in this case the living room of the occupant and the woman. The virtual environment can be configured to include animated 3D models of occupants and external persons based at least in part on sensor data and / or occupant data. The virtual environment can also use a preferred communication channel for external persons, which is determined for each person in the virtual environment. An attention score is intuitively represented, indicating the degree to which the occupant should pay attention to external traffic conditions.

[0088] Figure 3 The steps of a method for providing an interactive virtual environment within a vehicle according to various embodiments are schematically illustrated.

[0089] The method starts with step S10. In step S20, sensor data is acquired from at least one vehicle sensor. In step S30, the sensor data is processed to determine occupant data associated with an occupant inside the vehicle. In step S40, an interactive virtual environment is configured based on the occupant data. In step S50, the interactive virtual environment is displayed to the occupant on a vehicle display. The method ends in step S60.

[0090] Figure 4 A computing device 10 configured to execute a method for providing an interactive virtual environment within a vehicle is schematically illustrated according to various embodiments.

[0091] like Figure 4 As can be seen in the figure, the computing device 10 includes a processor 11 and a memory 12, wherein the memory 12 includes a program code, which, when executed by the processor 11, causes the processing device 10 to perform a method for providing an interactive virtual environment in a vehicle according to the following steps: receiving sensor data from at least one sensor 2, 3 on the vehicle, processing the received sensor data to determine occupant data related to an occupant inside the vehicle, configuring the interactive virtual environment based on the determined occupant data, and outputting the configured interactive virtual environment for display to the occupant on the vehicle display device 4.

[0092] The computing device 10 may be integrated within the vehicle 1 and utilize data from the internal sensors 2 and external sensors 3 to identify the occupants and customize the virtual environment experience accordingly. The computing device 10 facilitates communication between the occupants and other persons represented as avatars in the virtual environment. The computing device 10, together with the display 4 and the sensors 2, may also be referred to as a virtual environment system.

[0093] In summary, a technique for establishing an interactive virtual environment within a vehicle is provided. Sensor data from at least one sensor on the vehicle is collected. The sensor data is processed by an onboard computing device to obtain specific occupant data related to an occupant, including information about the occupant's presence, identity, location, or physical state. Using the occupant data, the computing device configures an interactive virtual environment that includes a digital avatar representing the occupant. The avatar is then displayed in the virtual environment to indicate the occupant's presence and whether communication with others is available. The virtual environment may include an avatar of another person, thereby facilitating real-time communication between them. The personalized and interactive virtual space and the avatars of other people are displayed to the occupant through an integrated display within the vehicle.

[0094] A method for providing an interactive virtual communication environment within a vehicle may use real-time sensor data. The method acquires data from one or more vehicle sensors and then processes the data to obtain real-time occupant-specific data. Using the occupant data, the method may dynamically configure a virtual environment displayed on a display device of the vehicle. Within the virtual environment, an avatar representing a vehicle occupant is generated based on the real-time sensor data and associated occupant information. The avatar acts as an indicator of the occupant's presence, attention, and identity within the virtual environment, thereby providing a more accurate and real-time indication of whether the occupant is available to interact, thereby reducing the hardware complexity of the communication platform within the vehicle.

[0095] Although the methods, computing devices, virtual reality systems, and vehicles for providing a virtual communication environment described herein have been described in the context of land vehicles, particularly automobiles, it should be understood that the application of the techniques and configurations is not limited to automobiles or land vehicles.

[0096] The techniques are applicable to a variety of vehicles and methods of transportation, including but not limited to airplanes, ships, trains, or other forms of transportation. The examples provided in the context of an automobile are intended to illustrate the concepts in a particular implementation, but the broad techniques may be applicable to other vehicles or moving objects used to transport people. Further adjustments or modifications may be required to optimize the methods and systems for different vehicles or environments, but the core technologies related to vehicle control, passenger monitoring, entertainment systems, and other functions may still provide value in those alternative implementations.

[0097] The description and drawings are illustrative only and do not limit the scope of the claims to any particular implementation or use. Although the disclosed technology has been described with respect to certain preferred embodiments, equivalents and modifications will occur to others skilled in the art upon reading and understanding this specification. The present disclosure includes all such equivalents and modifications and is limited only by the scope of the appended claims.

Claims

1. A method for providing an interactive virtual environment in a vehicle, comprising: - acquiring sensor data from at least one vehicle sensor; - processing the sensor data by a processing device to determine occupant data associated with an occupant within the vehicle; - configuring, by a processing device, an interactive virtual environment using said occupant data; as well as - displaying the interactive virtual environment to the occupants via a vehicle display.

2. The method of claim 1, wherein the at least one sensor is selected from the group consisting of an in-cabin camera, a fatigue detection sensor, a seat-embedded pressure sensor, a smartphone-enabled NFC or Bluetooth sensor, a digital key sensor, and an in-cabin radar sensor.

3. A method as claimed in any preceding claim, wherein processing the sensor data comprises detecting entry of the occupant into the vehicle, and the occupant data comprises an indication of the presence of an occupant in the cabin of the vehicle.

4. A method as claimed in any preceding claim, wherein processing the sensor data comprises determining an identity of the occupant, and the occupant data comprises identification data indicative of the identity of the occupant.

5. The method of any preceding claim, further comprising: - obtaining a 3D model representing a meeting area outside the vehicle and a 3D model representing the occupant based on the occupant data, and configuring the interactive virtual environment depicting the meeting area and the 3D model of the occupant.

6. The method of any preceding claim, further comprising: - notifying another person outside the vehicle with whom the occupant intends to communicate of the occupant's presence in the virtual environment.

7. The method of any preceding claim, further comprising: - obtaining a 3D model representing said other person, and configuring said interactive virtual environment using said 3D model of said other person.

8. The method of any preceding claim, wherein obtaining the 3D model of the other person comprises: - communicating at least a portion of said occupant data via a communication network; - in response to said communication, receiving said 3D model representing said other person via said communication network.

9. The method of claim 8, wherein the displayed interactive environment includes avatars derived from the 3D models of the occupant and the other person.

10. The method of one of claims 6 to 9, wherein the occupant is the driver of the vehicle and the other person is a family member or friend of the occupant located in an interior living area represented in the virtual environment as the meeting area.

11. The method according to one of claims 6 to 10, further comprising: - retrieving a preferred communication channel of the other person, the preferred communication channel enabling the other person to interact with the occupant within the interactive virtual environment of the vehicle; as well as - configuring said interactive virtual environment using said preferred communication channel.

12. The method of any preceding claim, further comprising: - acquiring additional sensor data from at least one vehicle sensor monitoring external traffic conditions around the vehicle; - processing the sensor data and the additional sensor data to determine an attention score that quantifies the degree of attention required by the occupant to respond to the external traffic condition; as well as -Configuring the interactive virtual environment to visually represent the attention score.

13. A method as claimed in any preceding claim, wherein the vehicle display device is selected from the group consisting of: -Integrated infotainment screen; - a head-up display (HUD) projected onto the windshield of the vehicle; -Digital instrument panel; - Rear seat entertainment screens; or - A detachable tablet or mobile device docked within the vehicle.

14. A computing device integrated in a vehicle, comprising a processing unit and a memory, wherein the memory comprises instructions which, when executed by the processor, cause the computing device to perform the following steps: - receiving sensor data from at least one sensor on the vehicle; - processing the received sensor data to determine occupant data relating to an occupant within the vehicle; - configuring the interactive virtual environment based on the determined occupant data; as well as - outputting the configured interactive virtual environment for display to the occupant on a vehicle display device.

15. A vehicle comprising the computing device of claim 14.

Images