Immersive vehicle-mounted virtual entertainment system and method

Abstract

The invention specifically discloses an immersive vehicle-mounted virtual entertainment system and method. The system comprises AR equipment, an out-of-vehicle information acquisition device, an in-vehicle passenger state acquisition device and a cabin system. The cabin system is in signal connection with the AR equipment, the outside information acquisition device and the inside passenger state acquisition device. The outside-vehicle information collection device is used for collecting outside-vehicle weather information, road information, biological information, building information and surrounding vehicle information so as to obtain outside-vehicle dynamic data. The in-vehicle passenger state collecting device is used for collecting physiological information, behavior emotional states and sound information of in-vehicle passengers so as to obtain in-vehicle passenger state data. The cockpit system is provided with a cockpit large model, and the cockpit large model generates a virtual image, a point cloud and a depth image according to the vehicle state data, the out-of-vehicle dynamic data and the in-vehicle passenger state data and transmits the virtual image, the point cloud and the depth image to the AR equipment so as to overlap virtual reality images and automatically adjust cockpit hardware. The immersive experience of passengers can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of in-vehicle entertainment equipment, and more specifically, to an immersive in-vehicle virtual entertainment system and method. Background Art

[0002] With the development of automobile technology, in-vehicle entertainment systems have gradually become an important part of improving users' driving experience. However, the current in-vehicle entertainment systems mainly focus on functions such as audio and video playback, navigation, and basic information display, which are difficult to meet the needs of modern passengers for rich and interactive entertainment experiences. At the same time, existing in-vehicle virtual reality (VR) or augmented reality (AR) systems are mostly used to convey driving information. For example, AR-HUD provides drivers with navigation guidance information and safety warnings, but pays less attention to passengers' entertainment needs. Therefore, it is of great significance to dynamically adjust the virtual reality screen according to the vehicle driving data and the occupant status to synchronize the virtual experience with the driving screen. Summary of the invention

[0003] The present invention provides an immersive in-vehicle virtual entertainment system and method, which solves the problems of the existing in-vehicle entertainment system in which driving data, driving pictures and cockpit hardware equipment cannot be effectively combined, the virtual scene lacks interaction related to the vehicle driving status, and there is little interactivity and poor immersion. The system can improve the interactivity, immersion and personalization of the in-vehicle virtual entertainment system and increase the immersive experience of the passengers.

[0004] To achieve the above objectives, the present invention provides the following technical solutions:

[0005] An immersive in-vehicle virtual entertainment system, comprising: an AR device, an external information collection device, an internal passenger status collection device, and a cockpit system;

[0006] The cockpit system is respectively connected to the AR device, the external information collection device and the internal passenger status collection device by signals;

[0007] The vehicle external information collection device is used to collect weather information, road information, biological information, building information and surrounding vehicle information outside the vehicle to obtain dynamic data outside the vehicle;

[0008] The vehicle occupant status collection device is used to collect the physiological information, behavioral emotional state and sound information of the vehicle occupants to obtain the vehicle occupant status data;

[0009] The cockpit system sets up a large cockpit model, and the large cockpit model generates virtual images, point clouds and depth images according to the vehicle status data, the external dynamic data and the internal occupant status data, and transmits them to the AR device to overlap the virtual reality screen and automatically adjust the cockpit hardware.

[0010] Preferably, the vehicle exterior information collection device comprises: an exterior radar, an exterior camera and an exterior light sensor;

[0011] The external radar is used to capture the driving status and position information of surrounding vehicles;

[0012] The external camera is used to capture weather information, road information, biological information and building information outside the vehicle;

[0013] The vehicle exterior light sensor is used to capture light change information under different weather and lighting conditions.

[0014] Preferably, the in-vehicle occupant status collection device comprises: an in-vehicle IMS camera and a microphone;

[0015] The in-vehicle IMS camera is used to identify the physiological information and behavioral emotional state of the occupants;

[0016] The microphone is used to collect voice information of the occupants.

[0017] Preferably, the cockpit system collects vehicle information in real time through a cockpit speed sensor, an acceleration sensor, a gyroscope, a steering angle sensor, a GPS receiver and a wheel speed sensor, and the vehicle information includes: vehicle speed, acceleration, turning angle and driving trajectory.

[0018] Preferably, the cockpit system is provided with an onboard map for collecting traffic conditions ahead of the vehicle and collecting surrounding map landmarks;

[0019] The traffic status ahead of the vehicle includes: real-time traffic flow and accident information;

[0020] The surrounding map landmarks include: landmark buildings and location coordinates along the way.

[0021] Preferably, the cockpit system is provided with a cockpit domain controller, and the cockpit hardware status is collected by the cockpit domain controller, and the cockpit hardware status includes: seat position and air conditioning setting.

[0022] The present invention also provides an immersive in-vehicle virtual entertainment method, using the above-mentioned in-vehicle virtual entertainment system, comprising:

[0023] Obtain vehicle status data, external dynamic data and internal passenger status data;

[0024] Identify and analyze the vehicle status data, the external dynamic data, and the internal occupant status data through a cockpit large model in the cockpit system, so as to generate a virtual image, a point cloud, and a depth image and transmit them to an AR device;

[0025] The AR device comprehensively analyzes the input virtual image, point cloud and depth image, and projects the virtual digital picture onto the front windshield in front of the co-pilot. After reflection, it enters the human eye, and then through optical alignment, the real picture seen by the human eye in front and the virtual picture reflected to the human eye are superimposed and integrated to obtain a real-time image that combines the human eye special effects animation with the real picture.

[0026] Preferably, it also includes:

[0027] The cockpit large model generates hardware adjustment requirements based on the vehicle status data, the external dynamic data and the in-vehicle occupant status data, and issues adjustment requirement instructions to music, air conditioning, audio, ambient lighting, seats and fragrance through the CAN bus.

[0028] Preferably, it also includes:

[0029] Through the external dynamic data and the vehicle status data, combined with the voice requirements input by the user and the user gesture operation, the virtual reality picture content is dynamically adjusted, and the adjustment requirements for the cockpit hardware are adjusted in real time.

[0030] Preferably, it also includes:

[0031] The cockpit large model is used to analyze the physiological information and behavioral emotional state of the occupants, and provide them with personalized entertainment content to suit different user preferences.

[0032] The present invention provides an immersive in-vehicle virtual entertainment system and method. The cockpit large model generates virtual images and cockpit hardware adjustment requirements based on vehicle status data, external dynamic data and in-vehicle occupant status data, and realizes the overlap of virtual and real images and automatic adjustment of cockpit hardware through AR equipment. The existing in-vehicle entertainment system solves the problems of failure to effectively combine driving data, driving images and cockpit hardware equipment, lack of interaction related to the driving status of the vehicle in the virtual scene, low interactivity and poor immersion. The system can improve the interactivity, immersion and personalization of the in-vehicle virtual entertainment system and increase the immersive experience of the passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below.

[0034] Figure 1 It is a schematic diagram of an immersive in-vehicle virtual entertainment system provided by the present invention.

[0035] Figure 2 It is a schematic diagram of an immersive in-vehicle virtual entertainment method provided by the present invention. DETAILED DESCRIPTION

[0036] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0037] The virtual scenes of the current in-vehicle entertainment system lack interaction related to the driving status of the vehicle, and there are problems of little interactivity and poor immersion. The present invention provides an immersive in-vehicle virtual entertainment system and method, which solves the problems that the existing in-vehicle entertainment system fails to effectively combine driving data, driving pictures and cockpit hardware equipment, and the virtual scenes lack interaction related to the driving status of the vehicle, and there are little interactivity and poor immersion. It can improve the interactivity, immersion and personalization of the in-vehicle virtual entertainment system and increase the immersive experience of the passengers.

[0038] like Figure 1 As shown, an immersive in-vehicle virtual entertainment system includes: an AR device, an external information collection device, an in-vehicle occupant status collection device and a cockpit system. The cockpit system is signal-connected to the AR device, the external information collection device and the in-vehicle occupant status collection device respectively. The external information collection device is used to collect weather information, road information, biological information, building information and surrounding vehicle information outside the vehicle to obtain dynamic data outside the vehicle. The in-vehicle occupant status collection device is used to collect physiological information, behavioral emotional state and sound information of the in-vehicle occupants to obtain in-vehicle occupant status data. The cockpit system is provided with a large cockpit model, and the large cockpit model generates virtual images, point clouds and depth images according to the vehicle status data, the external dynamic data and the in-vehicle occupant status data, and transmits them to the AR device to overlap the virtual reality screen and automatically adjust the cockpit hardware.

[0039] Specifically, the cockpit large model is based on vehicle status data, external dynamic data, and in-car passenger status data. It can dynamically adjust the virtual reality screen and the cockpit hardware status by acquiring and analyzing the vehicle's driving data and the status information of the co-pilot user in real time, ensuring that the virtual experience is synchronized with the driving screen. For example, when the vehicle accelerates or turns, the motion picture in the virtual scene will also change accordingly. This deep integration significantly enhances the passenger's sense of immersion. Traditional VR / AR systems and in-vehicle large-screen systems lack interaction related to the vehicle's driving status and are difficult to provide a realistic experience. The system has the following advantages: (1) Real-time data integration: By collecting vehicle driving data and co-pilot user status information in real time, the system can quickly respond to changes in the external environment and internal status. (2) Enhanced sense of reality: Using the large model to analyze data and generate AR dynamic virtual pictures that are synchronized with reality, it provides users with a more realistic immersive experience and significantly improves the riding experience, especially during long-distance travel. (3) Intelligent adjustment of cockpit hardware: Automatically adjust the air conditioning, seats, fragrances, etc. according to the virtual picture and user status to enhance riding comfort and immersion. (4) Personalized experience: By analyzing the physiological information and emotional state of the co-pilot user, the system can provide personalized entertainment content to adapt to different driving conditions and user preferences and meet the needs of different users. The system can improve the interactivity, immersion and personalization of the in-vehicle virtual entertainment system and increase the immersive experience of the passengers.

[0040] Furthermore, the vehicle exterior information collection device includes: an exterior radar, an exterior camera, and an exterior light sensor. The exterior radar is used to capture the driving status and location information of surrounding vehicles; the exterior camera is used to capture the exterior weather information, road information, biological information, and building information; and the exterior light sensor is used to capture the light change information under different weather and light conditions.

[0041] Furthermore, the in-vehicle occupant status collection device includes: an in-vehicle IMS camera and a microphone. The in-vehicle IMS camera is used to identify the physiological information and behavioral emotional state of the occupant; and the microphone is used to collect the occupant's voice information.

[0042] Specifically, the system acquires information about surrounding vehicles through an external radar, captures dynamic data such as weather, road conditions, biological information, and building information outside the vehicle through an external camera, identifies changes in light through an external light sensor, and then uses the internal IMS camera to identify the physiological information and behavioral emotional state of the co-driver. The microphone collects the voice information of the co-driver. The cockpit system acquires a large amount of real-time vehicle information, traffic conditions, map landmarks, and information on the status of the cockpit hardware. This information is input into the large cockpit model to generate virtual images and cockpit hardware adjustment requirements, and uses AR equipment to achieve the overlap of virtual and real images and automatic adjustment of the cockpit hardware.

[0043] Furthermore, the cockpit system collects vehicle information in real time through a cockpit speed sensor, an acceleration sensor, a gyroscope, a steering angle sensor, a GPS receiver and a wheel speed sensor, and the vehicle information includes: vehicle speed, acceleration, turning angle and driving trajectory.

[0044] Furthermore, the cockpit system is provided with an onboard map for collecting traffic conditions ahead of the vehicle and collecting surrounding map landmarks;

[0045] The traffic status ahead of the vehicle includes: real-time traffic flow and accident information;

[0046] The surrounding map landmarks include: landmark buildings and location coordinates along the way.

[0047] Furthermore, the cockpit system is provided with a cockpit domain controller, through which the cockpit hardware status is collected, and the cockpit hardware status includes: seat position and air conditioning setting.

[0048] It can be seen that the present invention provides an immersive in-vehicle virtual entertainment system. The cockpit large model generates virtual images and cockpit hardware adjustment requirements based on vehicle status data, external dynamic data and in-vehicle occupant status data, and realizes the overlap of virtual and real images and automatic adjustment of cockpit hardware through AR equipment. It solves the problems of the existing in-vehicle entertainment system in which driving data, driving images and cockpit hardware equipment are not effectively combined, the virtual scene lacks interaction related to the vehicle driving status, and there is little interactivity and poor immersion. It can improve the interactivity, immersion and personalization of the in-vehicle virtual entertainment system and increase the immersive experience of the passengers.

[0049] Accordingly, the present invention also provides an immersive in-vehicle virtual entertainment method, using the above-mentioned in-vehicle virtual entertainment system, comprising:

[0050] S1: Acquire vehicle status data, external dynamic data and internal passenger status data;

[0051] S2: identifying and analyzing the vehicle status data, the external dynamic data and the internal occupant status data through the cockpit large model in the cockpit system, so as to generate a virtual image, a point cloud and a depth image and transmit them to the AR device;

[0052] S3: The AR device conducts a comprehensive analysis of the input virtual image, point cloud and depth image, and projects the virtual digital image onto the front windshield in front of the co-pilot. After reflection, it enters the human eye, and then through optical alignment, the real image seen by the human eye in front and the virtual image reflected to the human eye are superimposed and integrated to obtain a real-time image that combines the human eye special effects animation with the real image.

[0053] Specifically, when the vehicle enters a high-speed driving state, the virtual screen will superimpose interesting auxiliary lines of acceleration and rapid movement around the vehicle; or when the vehicle turns, the perspective in the virtual screen will also fit the turning mark at the corner; corresponding explanatory images and text screens can also be superimposed next to landmark buildings to introduce characteristic buildings and attractions; similarly, virtual images of game props can be superimposed on the road ahead, and special effects will appear after the vehicle collides with the prop image, and it supports the use of props on the vehicle ahead, and the vehicle ahead will have corresponding superimposed special effects animations.

[0054] The method also includes: the cockpit large model generates hardware adjustment requirements based on the vehicle status data, the external dynamic data and the internal passenger status data, and issues adjustment requirement instructions to music, air conditioning, audio, ambient lights, seats and fragrances through the CAN bus.

[0055] The method also includes: dynamically adjusting the virtual reality screen content and adjusting the adjustment demand instructions for the cabin hardware in real time through the external dynamic data and the vehicle status data, combined with the voice demand input by the user and the user gesture operation.

[0056] The method also includes: analyzing the physiological information and behavioral emotional state of the occupants through a large cockpit model, and providing personalized entertainment content to the occupants to adapt to different user preferences.

[0057] Specifically, the cockpit model identifies and analyzes the transmitted vehicle status data and internal and external environment data, and generates virtual images, point clouds and depth images that are transmitted to the AR device; the generated hardware adjustment requirements are converted into CAN signals through the function cailing rules defined in the cockpit and transmitted to the cockpit hardware system, and then the cockpit hardware system issues instructions to various hardware, linking music, air conditioning, audio, ambient lights, seats, fragrances and other hardware. Through hardware such as external radar, cameras, and light sensors, the vehicle status data collected by the internal and external environments and the cockpit system are identified, and the voice requirements input by the microphone user and the user gesture operations captured by the camera are combined to dynamically adjust the virtual reality screen content and change the requirements for the cockpit hardware in real time. For example, in a strong light environment, the system automatically adjusts the brightness in the virtual reality; the user voice inputs the small plane that needs to move forward together in front, and the accompanying small plane picture immediately appears; the user gestures to the virtual cloud picture in front and waves his palm to disperse the clouds in front; or when the data shows that the vehicle is turning or entering a slope section, the seat automatically adjusts the angle and height to achieve a comfortable immersive experience.

[0058] It can be seen that the present invention provides an immersive in-vehicle virtual entertainment method. The cockpit large model generates virtual images and cockpit hardware adjustment requirements based on vehicle status data, external dynamic data and in-vehicle occupant status data, and realizes the overlap of virtual and real images and automatic adjustment of cockpit hardware through AR equipment. It solves the problems of the existing in-vehicle entertainment system in which driving data, driving images and cockpit hardware equipment are not effectively combined, the virtual scene lacks interaction related to the vehicle driving status, and there is little interactivity and poor immersion. It can improve the interactivity, immersion and personalization of the in-vehicle virtual entertainment system and increase the immersive experience of the passengers.

[0059] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the description and drawings, should be within the protection scope of the present invention.

Claims

1. An immersive in-vehicle virtual entertainment system, characterized in that: include: AR equipment, external information collection device, internal passenger status collection device and cockpit system; The cockpit system is respectively connected to the AR device, the external information collection device and the internal passenger status collection device by signals; The vehicle external information collection device is used to collect weather information, road information, biological information, building information and surrounding vehicle information outside the vehicle to obtain dynamic data outside the vehicle; The vehicle occupant status collection device is used to collect the physiological information, behavioral emotional state and sound information of the vehicle occupants to obtain the vehicle occupant status data; The cockpit system sets up a large cockpit model, and the large cockpit model generates virtual images, point clouds and depth images according to the vehicle status data, the external dynamic data and the internal occupant status data, and transmits them to the AR device to overlap the virtual reality screen and automatically adjust the cockpit hardware.

2. The immersive in-vehicle virtual entertainment system according to claim 1, characterized in that: The vehicle exterior information collection device includes: an exterior radar, an exterior camera and an exterior light sensor; The external radar is used to capture the driving status and position information of surrounding vehicles; The external camera is used to capture weather information, road information, biological information and building information outside the vehicle; The vehicle exterior light sensor is used to capture light change information under different weather and lighting conditions.

3. The immersive in-vehicle virtual entertainment system according to claim 2, characterized in that: The in-vehicle occupant status collection device includes: an in-vehicle IMS camera and a microphone; The in-vehicle IMS camera is used to identify the physiological information and behavioral emotional state of the occupants; The microphone is used to collect voice information of the occupants.

4. The immersive in-vehicle virtual entertainment system according to claim 3, characterized in that: The cockpit system collects vehicle information in real time through a cockpit speed sensor, an acceleration sensor, a gyroscope, a steering angle sensor, a GPS receiver and a wheel speed sensor. The vehicle information includes: vehicle speed, acceleration, turning angle and driving trajectory.

5. The immersive in-vehicle virtual entertainment system according to claim 4, characterized in that: The cockpit system is provided with an onboard map for collecting the traffic status in front of the vehicle and collecting surrounding map landmarks; The traffic status ahead of the vehicle includes: real-time traffic flow and accident information; The surrounding map landmarks include: landmark buildings and location coordinates along the way.

6. The immersive in-vehicle virtual entertainment system according to claim 5, characterized in that: The cockpit system is provided with a cockpit domain controller, through which the cockpit hardware status is collected, and the cockpit hardware status includes: seat position and air conditioning setting.

7. An immersive in-vehicle virtual entertainment method, using the in-vehicle virtual entertainment system according to any one of claims 1 to 6, characterized in that: include: Obtain vehicle status data, external dynamic data and internal passenger status data; Identify and analyze the vehicle status data, the external dynamic data, and the internal occupant status data through a cockpit large model in the cockpit system, so as to generate a virtual image, a point cloud, and a depth image and transmit them to an AR device; The AR device comprehensively analyzes the input virtual image, point cloud and depth image, and projects the virtual digital picture onto the front windshield in front of the co-pilot. After reflection, it enters the human eye, and then through optical alignment, the real picture seen by the human eye in front and the virtual picture reflected to the human eye are superimposed and integrated to obtain a real-time image that combines the human eye special effects animation with the real picture.

8. The immersive in-vehicle virtual entertainment method according to claim 7, characterized in that: Also includes: The cockpit large model generates hardware adjustment requirements based on the vehicle status data, the external dynamic data and the in-vehicle occupant status data, and issues adjustment requirement instructions to music, air conditioning, audio, ambient lighting, seats and fragrance through the CAN bus.

9. The immersive in-vehicle virtual entertainment method according to claim 8, characterized in that: Also includes: Through the external dynamic data and the vehicle status data, combined with the voice requirements input by the user and the user gesture operation, the virtual reality picture content is dynamically adjusted, and the adjustment requirements for the cockpit hardware are adjusted in real time.

10. The immersive in-vehicle virtual entertainment method according to claim 9, characterized in that: Also includes: The cockpit large model is used to analyze the physiological information and behavioral emotional state of the occupants, and provide them with personalized entertainment content to suit different user preferences.

Images