In-vehicle immersive visual output based on contextual environment

Abstract

The invention relates to in-vehicle immersive visual output based on a contextual environment. An in-vehicle immersive display system and method is provided, particularly a system and method for generating visual output for a vehicle cabin. The method includes determining a color hue profile, a texture style profile, and / or a driver emotion profile, where the color hue profile and / or the texture style profile are determined based on image data captured by an externally facing camera, and wherein the driver emotion profile is determined based on sensor data captured by a driver monitoring sensor; and determining a visual output based on the driver emotion profile, the color tone profile, and / or the texture pattern profile.

Description

Technical Field

[0001] The present invention relates to methods and systems for controlling the output of light generating devices such as electronic displays and lights (e.g., light emitting diodes (LEDs)) in a vehicle, and more particularly to coordinating the visual output of such light generating objects, including graphics / content selection and / or image / graphic image processing, such as modifying the color attributes of the visual output. Background Art

[0002] Today, certain display systems (such as those used in automotive cockpits) can include display systems that combine several color display devices and other ambient lighting sources / diffused lighting sources, and this can introduce undesirable color differences in the actual output or the perceived output. Various electronic displays can be used, such as, for example, one or more of liquid crystal displays (LCDs), light emitting diode (LED)-based displays (e.g., organic LEDs (OLEDs), LED LCDs), and projectors (e.g., LCD projectors, digital light processing (DLP) projectors). Such in-vehicle display systems also often include other ambient light sources or diffused light sources, such as LED arrays, LED strings, etc. In this sense, both OLED high-resolution television screens and devices including a single LED for generating visible light are considered "light sources" or "light generating devices".

[0003] Graphic lighting or display lighting refers to a light source that is part of an electronic display or a graphic interface, such as for backlighting a screen or for transmitting colored light. This lighting is typically used to convey information or enhance aesthetics through images, text, and other symbols or similar graphics. In an automotive background environment, this can include the backlighting of instrument clusters, infotainment screens, and control panels. Ambient lighting / diffused lighting refers to a light source that emits light without defined or distinct images, text, or symbols and can evenly and softly illuminate a space. In an automotive setting, ambient lighting is used in the vehicle cabin to provide soft indirect light that enhances visibility without causing glare or severe shadows, and can be used, for example, to provide mood lighting by allowing the user to make chromaticity modifications. Examples of ambient light / diffused light include LED strips or panels placed under the dashboard, in the door panels, or around the center console to provide soft and evenly distributed light to create a comfortable and safe driving environment.

[0004] Recently, more displays have been introduced into automotive applications to provide a better driving experience and effectively provide more information to the driver. A combination of various displays can provide an enhanced user experience for the driver and passengers; in particular, for example, a pillar-to-pillar display allows utilization of the entire width of the dashboard. In addition to combining two or more display screens, a sparse LED array can also be incorporated into the system. An in-cabin immersive system can include different LCD displays, a sparse LED array, an electronic mirror display, a rearview display, and in-vehicle lights. For example, U.S. Patent No. 11,620,099, issued on April 4, 2023, describes a display system for color matching multiple displays, such as those that can be used in an automobile.

[0005] However, even though color matching techniques, such as those described in U.S. Patent No. 11,620,099, improve the experience due to improved visual output consistency (e.g., visual output consistency between colors), it has been found that the immersion of the driver or other passengers can be improved by incorporating background environmental information, such as background environmental information related to the driver's emotional state or mental state (introverted background environment or inward perception) and / or the driver's environment (extroverted background environment or outward perception), as this helps to identify visual output that may be actively received by the driver or passenger.

[0006] Accordingly, a solution is provided that provides enhanced or increased immersion between the driver and the vehicle to provide a better driving experience for the driver, resulting in enhanced driving performance and reduced undesired driving errors. SUMMARY OF THE INVENTION

[0007] According to one aspect of the present disclosure, a method for generating visual output for a vehicle cabin is provided. The method includes: determining a color tone profile, a texture style profile, and / or a driver emotion profile. The color tone profile and / or the texture style profile are determined based on image data captured by an outward-facing camera, and the driver emotion profile is determined based on sensor data captured by a driver monitoring sensor. The method further includes: determining the visual output based on the driver emotion profile, the color tone profile, and / or the texture style profile.

[0008] According to various embodiments, the method may further include any one of the following features or any technically feasible combination of some or all of the following features:

[0009] - Determine the color tone profile, wherein the visual output is determined based on using a color transfer algorithm that generates output color information based on input color information, and wherein the input color information is based on the color information represented by the color tone profile within the image data or is generated based on the color information;

[0010] - The image data represents an image of a scene, and wherein the color information within the image data corresponds to the color information of one or more pixels within a determined region of the image;

[0011] - The determined region of the image is determined based on the output of a classifier, and wherein the classifier is used to classify portions of the image;

[0012] - The determined region of the image is pre-determined based on one or more pixel coordinates of the image;

[0013] - Determine the driver emotion profile, and wherein the driver emotion profile specifies one or more emotions selected from a plurality of pre-determined emotions, and wherein each emotion within the plurality of pre-determined emotions is associated with modification data indicating how the emotion affects the visual output;

[0014] - Determine the driver emotion profile, and wherein the driver emotion profile is a regression output represented by one or more tensors;

[0015] - Continuously obtain updated image data from the outward-facing camera, wherein each of the determining steps is performed using the updated image data to determine an updated visual output as the visual output for each iteration, and wherein the presentation of the visual output is continuously updated based on the updated visual output; and / or

[0016] - The visual output is an immersive visual output including a display output and a light output, wherein the display output is information indicating the output for a display screen, and wherein the light output is information indicating the output for an ambient light source or a diffused light source.

[0017] According to another aspect of the present disclosure, there is provided an in-vehicle immersive display system. The in-vehicle immersive display system includes: an electronic display mounted in a vehicle and configured to provide a display output; a light generating device mounted in the vehicle and configured to provide a light output; a camera mounted in the vehicle and configured to provide image data; and a controller mounted in the vehicle and configured to determine the display output and the light output based at least in part on the image data.

[0018] According to various embodiments, the in-vehicle immersive display system may further include any one of the following features or any technically feasible combination of some or all of the following features:

[0019] - The controller is further configured to: determine a color tone profile and a texture style profile based on the image data captured by the camera; determine a driver emotion profile based on the sensor data captured by the driver monitoring sensor; and determine a visual output for the immersive display system based on the color tone profile and the texture style profile, wherein the visual output is displayed by the immersive display system and consists of the display output and the light output;

[0020] - The driver emotion profile specifies one or more emotions selected from a plurality of pre-determined emotions, and each emotion in the plurality of pre-determined emotions is associated with modification data indicating how the emotion affects the visual output;

[0021] - The driver emotion profile is a regression output represented by one or more tensors;

[0022] - The display output and the light output are determined based on using a color transfer algorithm that generates output color information based on input color information, and wherein the input color information is based on the color information within the image data or is generated based on the color information;

[0023] - The image data represents an image of a scene, and the color information within the image data corresponds to the color information of one or more pixels within a determined region of the image;

[0024] - The determined region of the image is determined based on the output of a classifier, and the classifier is used to classify parts of the image;

[0025] - The determined region of the image is pre-determined based on one or more pixel coordinates of the image;

[0026] - The controller is further configured to continuously obtain updated image data from the outward-facing camera, wherein the visual output is displayed by the immersive display system and consists of the display output and the light output, wherein the determination operations are each performed using the updated image data to determine an updated visual output as the visual output for each iteration, and wherein the display of the visual output is continuously updated according to the updated visual output; and / or

[0027] - The visual output is displayed by the immersive display system and consists of the display output and the light output, and wherein the visual output is information indicating an output for a display screen and an output for an ambient light source.

[0028] According to another aspect of the present disclosure, there is provided an in-vehicle immersive display system. The in-vehicle immersive display system includes: an electronic display installed in a vehicle and configured to provide a display output; a diffused light generating device installed in the vehicle and configured to provide a light output; an outward-facing camera installed in the vehicle and configured to provide image data; a driver monitoring sensor installed in the vehicle and configured to provide driver monitoring sensor data; and a controller installed in the vehicle and configured to determine the display output and the light output at least in part based on the image data provided by the outward-facing camera and the driver monitoring sensor data provided by the driver monitoring sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Preferred exemplary embodiments will be described below in conjunction with the accompanying drawings, where like reference numerals represent like elements, and wherein:

[0030] Figure 1 is a block diagram showing an in-vehicle immersive display system according to an embodiment;

[0031] Figure 2 is a flowchart showing a method for generating a visual output for a vehicle cabin according to an embodiment;

[0032] Figure 3 is a block diagram showing an in-vehicle immersive display framework and a processing flow according to an embodiment; and

[0033] Figure 4 is a block diagram depicting an arrangement of a display device and other light sources used as part of an in-vehicle immersive display system according to an embodiment. DETAILED DESCRIPTION

[0034] A system and method for generating visual output for use by a driver or other vehicle passengers is provided to enhance in-cabin immersion and / or otherwise improve the in-vehicle experience. The term "immersion" when used in the context of a vehicle (e.g., "in-vehicle immersion", "in-cabin immersion") refers to the sensory alignment of an individual with the vehicle, including sensory alignment from a visual perspective and / or an auditory perspective. Nowadays, the cabin of a vehicle typically has many different light sources, such as displays for infotainment units, electronic mirror / e-mirror displays, instrument cluster lights / displays, light-emitting diode (LED) arrays, other LED lights, etc. To enhance the in-vehicle experience, the light output can be coordinated, such as where the same, similar, or complementary colors are used for the light output of multiple different light sources, which light output is referred to herein as visual output. This coordination of light or content / display screens is used to enhance in-vehicle immersion.

[0035] According to an embodiment, the visual output generated by the system is an immersive visual output, which is a visual output formed by using multiple light sources within the vehicle to generate light simultaneously (i.e., at the same time); for example, a display generates a first visual output and an LED array generates a second visual output, and the first visual output and the second visual output together constitute an immersive visual output; however, in other embodiments, other light sources can be used to generate the immersive visual output, such as the light sources mentioned above.

[0036] According to an embodiment, an auditory output is generated by the system and, in the embodiment, is output simultaneously with the immersive visual output, thereby providing an immersive multi-sensory output, more specifically, providing an immersive audio-visual output. For example, the auditory output is generated by using speakers; however, various different audio output devices can be used.

[0037] According to an embodiment, the in-vehicle immersion is enhanced by coordinating the visual output with the vehicle's surrounding environment and / or the driver's emotional state, by connecting the driver (or other vehicle passengers) to the visual output.

[0038] According to an embodiment, an in-vehicle immersion enhancement framework based on the driving environment of a vehicle and the emotional state of a driver is provided. By incorporating such information into the display of in-vehicle displays and lights or other visual outputs, the driving performance and mental state of a driver can be improved. Specifically, in an embodiment, a framework is used to generate new content (and / or revised content) for immersive displays and systems, thereby providing enhanced immersion across different displays, systems, and driving environments. In at least one embodiment, content with enhanced immersion is generated based on color transfer / style transfer, driving environment analysis, and driver emotional state identification. More specifically, according to an embodiment, a framework is provided that enhances in-cabin / in-vehicle immersion by providing a consistent hue and texture style across different displays, providing adjusted content according to the driving environment, and / or providing appropriate color adjustments in in-cabin immersive displays and systems according to the emotional state of the driver.

[0039] Reference Figure 1 , there is shown an in-vehicle immersive display system 10, which has a controller 12, an external-facing camera 14, a driver-facing camera serving as a driver monitoring sensor 16, an electronic display 18, an electronic mirror display 20, instrument cluster lights 22, and sparse LEDs 24. The controller 12 has at least one processor 26 and a memory 28 storing computer instructions that, when executed by the at least one processor 26, cause functions belonging to the controller 12, such as one or more steps of a method, to be performed. Figure 1 The in-vehicle immersive display system 10 is one embodiment, and in other embodiments, the in-vehicle immersive display system may include one or more other components in addition to or instead of one or more of the components of system 10.

[0040] The controller 12 is used to control the visual output of the in-vehicle immersive display system 10, such as the graphics displayed on the display 18 or the color and / or brightness of the light emitted by the sparse LEDs 24. As described above, in at least an embodiment, the computer instructions stored on the memory 28 direct the controller 12 to perform one or more of the steps of the method discussed below. The controller 12 is communicatively coupled to the external-facing camera 14 and the driver-facing camera 16 to receive image data from the cameras 14, 16. For example, the controller 12 is also communicatively coupled to the output devices 18-24, such as via a controller area network (CAN) bus or a wireless local area network (WLAN) connection. However, any suitable communication connection or link may be used.

[0041] Although the controller 12 is described as a single device, it should be understood that multiple computers or devices can be used as the controller 12, which are configured together to perform the methods described herein and any other functions attributed to the controller 12. It should also be understood that computer instructions can be stored on different physical memory devices and / or executed by different processors or computers of the controller 12, together causing the execution of the methods and functions discussed herein.

[0042] Any one or more of the processors discussed herein can be implemented as any suitable electronic hardware capable of processing computer instructions and can be selected based on the application for which it will be used. Examples of processor types that can be used include central processing unit (CPU), graphics processing unit (GPU), field programmable gate array (FPGA), application specific integrated circuit (ASIC), microprocessor, microcontroller, etc. Any one or more of the memories discussed herein can be implemented as any suitable type of non-transitory computer-readable memory capable of storing data or information in a non-volatile manner and in electronic form such that the stored data or information can be consumed by a processor. The memory can be any one of a variety of different types of electronic memory and can be selected based on the application for which it will be used. Examples of memory types that can be used include disk or optical disk drives, ROM (read only memory), solid state drive (SSD) (including other solid state storage devices such as solid state hybrid drive (SSHD)), other types of flash memory, hard disk drive (HDD), non-volatile random access memory (NVRAM), etc. It should be understood that any one or more of the computers or controllers discussed herein can include other memories, such as volatile RAM used by the processor and / or multiple processors.

[0043] In one embodiment, at least one processor 26 includes a central processing unit (CPU) and a graphics processing unit (GPU) (or even a tensor processing unit (TPU)), each of the central processing unit and the graphics processing unit being used to perform different functions of the controller 12. For example, the GPU is used for image signal processing and inference of neural networks (or any similar machine learning model), and for any training, such as online training for adaptive learning implemented after initial deployment; on the other hand, the CPU can perform other functions. Of course, this is only an example of a specific implementation of the controller 12, as will be understood by those skilled in the art, other hardware devices and configurations can be used, which generally depends on the specific application in which the controller 12 is used.

[0044] The outward-facing camera 14 is an example of an environmental sensor and a visible light sensor. As used herein, a visible light sensor is a light sensor that captures visible light, which is represented as an array of pixels that together form a visible light image. A visible light sensor is a camera that captures and represents a scene using a visible light color space or visible light color domain (e.g., RGB). According to an embodiment, the visible light sensor is a digital camera, such as a digital camera employing a CMOS (complementary metal oxide semiconductor) sensor, a CCD (charge-coupled device) sensor, or a Foveon sensor.

[0045] The outward-facing camera 14 is configured to capture a visible light image, which is an image representing the visible light captured by the image sensor. The outward-facing camera 14 can be any of a variety of suitable camera types, such as a digital single-lens reflex (DSLR) camera, a mirrorless camera, a dash cam (e.g., Garmin Dash Cam56 TM ), a compact digital camera, a pinhole camera, etc. The outward-facing camera 14 is a camera and is "outward-facing" because the field of view of the camera faces and captures an area outside the vehicle, such as the road on which the vehicle is traveling, the skyline, and / or the sky. The outward-facing camera 14 is used to capture visible light image data, and the visible light image data is used to determine, for example, the visual output of the immersive display system 10 by determining a color tone profile that sets one or more tones to be used in the content displayed on the electronic display 18 or the color or brightness of the diffused light emitted by the sparse LEDs 24. In an embodiment, the visible light image data is also displayed for the driver or passenger of the vehicle, such as by displaying a live video stream of the visible light image data captured by the outward-facing camera 14 on the display 18. Although only a single environmental sensor is included in the depicted embodiment, in other embodiments, the system 10 includes multiple environmental sensors, such as multiple outward-facing cameras.

[0046] The driver monitoring sensor 16 is used to determine the driver's mood or other mental state, and can be a wearable or fixed sensor physically coupled to the driver (e.g., a heart rate sensor in a smartwatch worn by the driver) or a wearable or fixed sensor spaced apart from the driver, such as a camera facing the driver or a vehicle interior microphone. In modern vehicles, a complex set of sensors, such as facial recognition and analysis technology, voice analysis tools, biometric sensors, electroencephalography (EEG), eye tracking systems, movement and posture analysis, and wearable technology, can be employed for driver monitoring. In an embodiment, these sensors work together to detect and interpret various emotional and mental states of the driver; however, in other embodiments, a single sensor may be sufficient to determine the driver mood profile. For example, a camera for facial recognition and analysis can pick up subtle changes in the driver's expression, indicating stress or drowsiness, while voice analysis can capture changes in tone and language patterns suggesting frustration or fatigue. Heart rate monitors and other biometric sensors embedded in the vehicle (or worn by the user and communicatively coupled to the vehicle) can track physiological responses, such as an increased heart rate, signaling anxiety or excitement. An eye tracking system monitors the driver's gaze and pupil dilation to detect distraction or reduced alertness. Observing the driver's body language through movement and posture analysis can reveal signs of restlessness or drowsiness. Additionally, data from wearable technology, such as a smartwatch, can inform the driver's overall health and stress level, thus influencing their driving behavior. In an embodiment, one or more of the driver monitoring sensors can be integrated with machine learning algorithms, which can enable a comprehensive understanding of the driver's state. Although only a single driver monitoring sensor is included in the depicted embodiment, in other embodiments, the system 10 can include multiple driver monitoring sensors.

[0047] In other embodiments, other driver monitoring devices may be used in addition to visible light image processing. For example, Babusiak B, Hajducik A, Medvecky S, Lukac M, Klarak J, "Design of Smart Steering Wheel for Unobtrusive Health and Drowsiness Monitoring". Sensors. 2021;21(16):5285 teaches a smart steering wheel equipped with an electrocardiograph, a pulse oximeter, and an inertial measurement unit that monitors the driver's heart rate, blood oxygen, and movement patterns. These physiological parameters can provide insights into the driver's emotional state, such as stress or relaxation levels. Convolutional and recurrent neural networks can be used to process this data, for example, potentially identifying patterns associated with specific emotional states.

[0048] The electronic display (or "display") 18 is used to present graphical content and / or text content to the user, and the display 18 can be any of a variety of display devices, such as a liquid crystal display (LCD), a light-emitting diode (LED)-based display (e.g., organic LED (OLED), LED LCD), and a projector (e.g., LCD projector, digital light processing (DLP) projector). The electronic display 18 is communicatively coupled to the controller 12 and receives content from the controller 12, and the content is displayed. The output content is an example of a visual output.

[0049] The electronic mirror display 20 can be used as part of an electronic mirror (e-mirror / E-mirror) system that utilizes display technology to replace or enhance a conventional mirror. The electronic mirror system mainly includes an outward-facing camera (such as the outward-facing camera 14) that relays real-time visual information to one or more display screens of the electronic mirror display 20. The outward-facing camera is located in the vehicle and is arranged to be easily within the driver's field of view when in the driver's position. Various display technologies can be employed in the electronic mirror display 20, for example, such as an LCD or OLED display. The electronic mirror display 20 is communicatively coupled to the controller 12 and receives content from the controller 12, and the content is displayed as a visual output.

[0050] The instrument cluster 22 is used to provide important driving information to the driver, such as speed, fuel level, engine temperature, etc., and the instrument cluster is often an instrument panel component integrated into the instrument panel. The instrument cluster 22 provides at least a part of the driving information to the driver through a visual output, and the visual output may include LEDs for backlighting or an electronic display with a display screen. In some embodiments, the instrument cluster 22 provides a visual output through a combination of different light sources. For example, ambient lighting is used for the first instrument reading and an electronic display is used for the second instrument reading. The instrument cluster 22 is communicatively coupled to the controller 12, receives content from the controller 12, and the content is displayed as a visual output. Although the instrument cluster 22 may include an electronic display, in this embodiment, the electronic display 18 is separated from the instrument cluster 22 and any display included as part of it.

[0051] The sparse LED 24 is an example of a diffused light generating device, and in this embodiment, the sparse LED is a light in the vehicle interior (such as in the passenger compartment of the vehicle) and is used to output colored ambient light. The sparse LED 24 can be an RGB LED (red LED, green LED, blue LED), an OLED, or a quantum dot LED (QLED). However, different from the display 18, the sparse LED generally generates diffused light and is not arranged to present distinguishable content on a pixel matrix. The sparse LED 24 is communicatively coupled to the controller 12, receives content from the controller 12, and the content is displayed as a visual output. Although the instrument cluster 22 may include sparse LEDs, in this embodiment, the sparse LED 24 is separated from the instrument cluster 22 and any sparse LED included as part of it.

[0052] Reference Figure 2 , an embodiment of a method 200 for generating a visual output for a vehicle compartment is shown. In at least one embodiment, the method 200 is performed by the in-vehicle immersive display system 10. Although the steps of the method 200 are discussed as being performed in a specific order, it should be understood that the steps of the method 200 can be performed in any technically feasible order, such as where step 220 is performed simultaneously with step 210 or before step 210.

[0053] Method 200 begins at step 210, where a color tone profile and a texture style profile are determined based on image data captured by an externally facing camera; that is, based on the image data captured by the externally facing camera, a color tone profile is determined, and based on the image data captured by the externally facing camera, a texture style profile is determined. The color tone profile refers to color or hue information, such as chroma, luminance, and saturation, and can be represented by one or more values (or ranges of values) for each of these tonal attributes. The texture style profile refers to a set of visual and tactile characteristics that collectively define and standardize a particular texture style, and the profile can include attributes such as pattern repeat, surface roughness or smoothness, reflectivity, and tactile quality.

[0054] In an embodiment, the externally facing camera 14 captures visible light image data of an external area of the vehicle. This area within the field of view (FOV) of the externally facing camera 14 can include the sky or other environmental space (e.g., the ceiling of a tunnel) above the road on which the vehicle is traveling, referred to as the "overhead area" (although the overhead area within the FOV at any given time may not be directly above the vehicle, but rather, for example, in front of the vehicle). In another example, the externally facing camera 14 captures visible light images of an area on the road side and / or on the road, which is referred to as the "road area", and the road side is specifically referred to as the "road side area", and the road area corresponding to the portion on the road (i.e., the non-road side area) is referred to as the "on-road area".

[0055] In an embodiment, the portion of the image corresponding to the overhead area and / or the road side area (rather than the entire image) is used to determine the color tone profile and / or the texture style profile. In another embodiment, the on-road area (rather than the road side area) is used to determine the color tone profile and / or the texture style profile. In other embodiments, the entire image can be used to determine the color tone profile and / or the texture style profile.

[0056] In one example, when the externally facing camera 14 captures an icing scene, a texture that mimics the appearance of snow and ice, having reflectivity, a crystal structure, and a sense of smoothness or gloss, is selected to be used in or as the texture style profile. In this example, the color tone profile includes white, blue, and other cool tones. And, in another example, when the externally facing camera 14 captures a sunny scene, a smoother and more diffusive texture (with a soft focus to convey a gentle sunlight quality and a peaceful atmosphere) is selected to be used in or as the texture style profile. In this example, the color tone profile includes warm and bright colors, such as yellow and orange, to create a feeling of warmth and light. Method 200 continues to step 220.

[0057] In step 220, based on the sensor data captured by the driver monitoring sensor, a driver emotion profile is determined. As discussed above, the driver monitoring sensor 16 is a sensor for capturing information about the driver to enable aspects of the driver's emotional state to be inferred or otherwise determined. The driver monitoring sensor 16 is a driver-facing camera that captures visible light images of the driver's face and / or other parts of the driver's body, and then processes the captured visible light images to determine the driver's emotion, such as whether the driver is happy, calm, anxious, or enraged. Method 200 proceeds to step 230.

[0058] In step 230, a visual output for the immersive display system is determined based on the color hue profile and the texture style profile. Refer to Figure 3 The determination is discussed, Figure 3 An embodiment of an immersive display frame 300 within a vehicle is depicted. The frame 300 includes a driving environment analyzer module 310, a driver emotion analyzer module 320, a style / color transfer module 330, and a seamless transition compensation module 340. Generally, each of the modules 310 - 340 is implemented using a processor and a memory, where the processor executes computer instructions that, when executed by the processor, cause the functions attributed to the respective modules 310 - 340 to be performed.

[0059] The driving environment analyzer module 310 constructs a color hue profile Pc and a texture style profile Pt based on the captured images from a camera (such as the outward-facing camera 18), which can be a front driving vision camera, a rear vision camera, or an electronic mirror camera. The color hue profile Pc is constructed by analyzing the color statistics of the environmental image (e.g., the mean and variance of each channel in the Lab color space). The texture style profile Pt is constructed by analyzing the local and global shape and texture of the environmental image. Of course, in other embodiments, other techniques can be used to determine the color hue profile and the texture style profile.

[0060] The driver emotion analyzer module 320 constructs a driver emotion profile Pe based on the output images of the in-cabin monitoring camera. The driver emotion profile Pe is obtained via image analysis and machine learning algorithms. In this embodiment, a two-dimensional representation of the driver emotion profile is made in terms of valence and arousal. Valence refers to whether the emotion is more positive or more negative, while arousal refers to the amount of activation in the emotion.

[0061] In an embodiment, the driver emotion profile is represented as a tensor matrix, which can represent emotion attributes or emotion characteristics. Here, the driver emotion profile is a regression output represented by one or more tensors. In this case, the regression output implies that the emotion profile is represented as a continuous output rather than a discrete category. The output can be represented by one or more tensors, where each tensor is a multi-dimensional array containing elements representing specific emotion attributes or parameters. In at least one embodiment, the tensor matrix, together with the input image, can be fed into a neural network to generate a modified image. Then, the neural network will generate an output image, and the neural network can be any type of machine learning model capable of processing this data (e.g., a convolutional neural network). The output image is modified based on the emotion represented in the tensor matrix, which incorporates the emotion into the image. In another embodiment, the emotions are discretely classified, which means that each emotion is clearly classified, rather than using a continuous spectrum of emotions represented in the tensor matrix. Then, the input image is modified using the associated information related to each discrete emotion category. This can involve applying specific filters or transforms corresponding to each emotion category. These two embodiments allow emotion data to be incorporated into image processing via a neural network, but these two embodiments handle the representation of emotions in different ways. In an embodiment, the driver emotion profile includes values each of which can be associated with one of a plurality of predetermined emotions. And, in an embodiment, each predetermined emotion is associated with modification data indicating how the emotion affects the visual output, such as a specific set of chromaticities, a brightness range, or an increment amount to be used or not used, etc.

[0062] The color / style transfer module 330 adjusts the color and / or style of the input content based on the driving environment profile Pc, Pt, and the emotion state profile Pe. In one embodiment, the color of the input content is adjusted via a color transfer algorithm based on the driving environment color hue profile Pc, while the style of the input content is adjusted via a style transfer algorithm based on the driving environment texture style profile Pt.

[0063] Color transfer algorithms are techniques used in computer graphics and image processing to apply the color characteristics of one image to another. This is typically used to make images appear as if they were taken under similar lighting conditions, or to achieve a specific artistic effect. Color transfer algorithms generally include: color space conversion, statistical analysis of color information, color mapping based on mean / standard deviation, and color space reconversion. In one embodiment, the color transfer algorithm used is the color transfer algorithm described in "Texture-Aware emotional Color Transfer Between Images" by S. Liu and M. Pei, IEEE Access, Vol. 6, pp. 31375-31386, 2018, doi: 10.1109 / ACCESS.2018.2844540. For example, this document describes a framework for emotion-based color transfer in images. The method includes three main steps. First, if the input is a reference image, the main color and texture features of the image are extracted, and using the proposed emotion calculation model, the target emotion coordinate values in the emotion scale are calculated using the main color and texture features. If the input is an emotion word, the most similar landmark word is found in the database using a semantic similarity algorithm, and then this most similar landmark word is considered the target emotion. The next step involves searching an emotion database to find the best-matching target emotion. These databases are built using theoretical and empirical concepts from art theory, which includes color models, corresponding emotion coordinates, emotion words, and hue numbers (HN) and color numbers (CN). Then, the closest color combination is obtained from one of the model databases in the model database. This color transfer framework can be used for texture-aware emotional color transfer, which changes the color of an image to match the desired emotion calculated from a reference image or an emotion word. In other embodiments, other color transfer algorithms can be used.

[0064] Style transfer algorithms are typically used to interpolate between two images in such a way that one image adopts the style of another. This technique is widely used in digital technology and image processing to generate desired visual effects, such as for reproducing the appearance and feel of the surrounding vehicle environment (e.g., snowy, desolate, urban). Style transfer involves using a convolutional neural network (CNN) to merge the content features of one image with the style features of another, and optimizing through an iterative process like gradient descent to create a new image that combines the original content with the artistic style of the second image.

[0065] In one embodiment, the style transfer algorithm used is the style transfer algorithm set forth in "Style Transfer Via Texture Synthesis" by M. Elad and P. Milantar (IEEE Transactions on Image Processing, Vol. 26, No. 5, pp. 2338 - 2351, May 2017, doi: 10.1109 / TIP.2017.2678168). According to Elad et al., the proposed style transfer algorithm uses a pre - trained CNN to apply the artistic style (style reference) of one image to another image (content image) while preserving its content. The algorithm involves three images: the content, the style reference, and the generated image, which is initially a copy of the content image. The algorithm uses the CNN to extract feature representations from the two images. High - level details are extracted from the content image, while texture and color are extracted from the style image. The style is captured using the Gram matrix, which is a mathematical representation of the correlations between different features in an image. This is done at each layer of the network to capture details at different levels. The algorithm defines a loss function that includes a content loss (the content difference between the generated image and the content image) and a style loss (the style difference between the generated image and the style image). The algorithm aims to minimize this loss function using backpropagation and gradient descent, thereby iteratively updating the generated image until convergence. The final output is an image that combines the content of the content image with the style of the style image. Although computationally intensive due to the use of deep learning and the iterative process, the results are visually impressive. In other embodiments, other style transfer algorithms may be used.

[0066] It also configures the emotional state profile Pc and adjusts the color of the input content in the direction of enhancing the driver's positive emotions or alleviating the driver's negative emotions. Based on the relationship between emotion and color ("Color and emotion: effects of hue, saturation, and brightness", Psychological Research, 2018), color adjustment is performed in the valence and arousal space (Russell, J.: "A circumplex model of affect". Journal of personality and social psychology, 39(6), 1161-1178 (1980)). For example, arousal increases from blue and green to red, and for saturated and bright colors, the valence level is the highest. As a more specific example, to suppress anger, the color hue is adjusted to a darker blue (dark green) color that is less saturated than saturated red; and to enhance happiness, the color hue is adjusted to saturated orange / yellow / pink.

[0067] Due to hardware physical boundaries, different resolutions, and different color gamuts, transitions across different displays in the in-cabin immersive display and system can be seen. The seamless transition compensation module 350 addresses this issue by applying color correction and progressive blur compensation. For example, U.S. Patent No. 11,620,099 issued on April 4, 2023, describes a display system and method for color matching multiple displays (such as displays that can be used in a vehicle); the description of the display system and method in U.S. Patent No. 11,620,099 is hereby incorporated by reference in its entirety and, to the extent that description is not inconsistent with the description herein, is attributed to the embodiments of the present disclosure.

[0068] Visual output can be provided to one or more display devices, such as one or more of the following: electronic display 18, electronic mirror display 20, instrument cluster 22, and / or sparse LED 24. Then, method 300 ends.

[0069] Method 300 can be continuously executed to continuously receive new image data and / or driver monitoring sensor data from the outward-facing camera 14 and then determine the updated visual output, which can be a relatively subtle change (e.g., a minor chromaticity change in the background graphics) or can be a more significant change (e.g., selecting a new main foreground graphic for display).

[0070] Method 300 can be executed continuously to generate immersive visual output for an immersive display subsystem, which refers to a light generating device for outputting visual output for an in-vehicle immersive display system.

[0071] Reference Figure 4 , an in-vehicle immersive display subsystem 400 is shown, which includes various light generating devices, including a display device and an ambient / diffuse device. In particular, the in-vehicle immersive display subsystem 400 includes a left electronic mirror (“electronic mirror” or “e-mirror”) 402, a right electronic mirror 404, an instrument cluster 406, a central information display (CID) 408, a first LED device 410, a second LED device 412, and a third LED device 414. The in-vehicle immersive display subsystem 400 can be used as part of an in-vehicle immersive display system 10; for example, the CID 408 corresponds to the electronic display 18, the electronic mirrors 402, 404 correspond to the electronic mirror displays 20, the instrument cluster 406 corresponds to the instrument cluster lights 22, and the LED devices 410-414 correspond to the sparse LEDs 24. Of course, in other embodiments, other in-vehicle immersive display subsystems with different devices, arrangements, and / or other characteristics can be used, as the in-vehicle immersive display subsystem 400 is merely one embodiment.

[0072] It should be understood that the foregoing description is one or more embodiments of the present invention. The present invention is not limited to the specific embodiments disclosed herein, but is only defined by the following appended claims. Additionally, the statements included in the foregoing description relate to the embodiments disclosed in the present invention and should not be construed as limiting the scope of the present invention or the definition of the terms used in the claims, unless the terms or phrases are expressly defined above. Various other embodiments and various changes and modifications to the embodiments disclosed in the present invention will become apparent to those skilled in the art.

[0073] As used in this specification and the claims, the words “enhanced,” “enhancing,” and other forms thereof should not be construed as limiting the present invention to any specific type or manner of visual presentation or visual output, but are generally used to facilitate understanding of the above technologies and, in particular, to convey that such technologies are intended to introduce visual aspects into the cabin or in-vehicle environment to provide a more positively acceptable environment for the driver or other passengers.

[0074] As used in this specification and the claims, the terms "e.g. / for example", "for instance", "such as", and "like", and the verbs "comprising", "having", "including", and their other verb forms, when used in conjunction with a list of one or more elements or other items, are each to be construed as open-ended, meaning that the list should not be regarded as excluding other additional elements or items. Other terms are to be construed in their broadest reasonable sense unless they are used in a context that requires a different interpretation. Additionally, the term "and / or" shall be construed as inclusive OR. Thus, for example, the phrase "A, B, and / or C" shall be construed to cover all of the following: "A"; "B"; "C"; "A and B"; "A and C"; "B and C"; and "A, B, and C".

Claims

1. A method of generating a visual output for a vehicle cabin, the method comprising the steps of: determining a color tone profile, a texture pattern profile, and / or a driver emotion profile, wherein the color tone profile and / or the texture pattern profile are determined based on image data captured by an outward-facing camera, and wherein the driver emotion profile is determined based on sensor data captured by a driver monitoring sensor, wherein the driver monitoring sensor is a sensor used to capture information about a driver to enable aspects of the driver's emotional state to be inferred or otherwise determined; as well as Based on the driver emotion profile, the color tone profile, and / or the texture pattern profile, a visual output is determined.

2. A method according to claim 1, wherein the method includes determining the color tone profile, wherein the visual output is determined based on using a color transfer algorithm, the color transfer algorithm generates output color information based on input color information, and wherein the input color information is based on color information represented by the color tone profile within the image data, or is generated based on the color information.

3. The method of claim 2, wherein the image data represents an image of a scene, and wherein the color information within the image data corresponds to color information of one or more pixels within the determined area of ​​the image. 4 . The method of claim 3 , wherein the determined region of the image is determined based on an output of a classifier, and wherein the classifier is used to classify the portion of the image. The method of claim 3 , wherein the determined region of the image is predetermined based on one or more pixel coordinates of the image.

6. A method according to claim 1, wherein the method includes determining the driver emotion profile, and wherein the driver emotion profile specifies one or more emotions selected from a plurality of predetermined emotions, and wherein each emotion in the plurality of predetermined emotions is associated with modification data indicating how the emotion affects the visual output. 7 . The method of claim 6 , wherein the method includes determining the driver emotion profile, and wherein the driver emotion profile is a regression output represented by one or more tensors.

8. The method according to claim 1, further comprising the steps of: Updated image data is continuously obtained from the outward-facing camera, wherein the determining steps are each performed using the updated image data to determine an updated visual output as the visual output for each iteration, and wherein the presentation of the visual output is continuously updated based on the updated visual output.

9. The method of claim 1, wherein the visual output is an immersive visual output including a display output and a light output, wherein the display output is information indicating an output for a display screen, and wherein the light output is information indicating an output for an ambient light source or a diffuse light source.

10. An in-vehicle immersive display system, the in-vehicle immersive display system comprising: an electronic display mounted in the vehicle and configured to provide a display output; a light generating device mounted in the vehicle and configured to provide a light output; a camera mounted in the vehicle and configured to provide image data; and A controller is mounted in the vehicle and is configured to determine the display output and the light output based at least in part on the image data.

11. An in-vehicle immersive display system according to claim 10, wherein the controller is further configured to: determine a color tone profile and a texture style profile based on image data captured by the camera; determine a driver emotion profile based on sensor data captured by a driver monitoring sensor; and determine a visual output for an immersive display system based on the color tone profile and the texture style profile, wherein the visual output is displayed by the immersive display system and consists of the display output and the light output.

12. An in-vehicle immersive display system according to claim 11, wherein the driver emotion profile specifies one or more emotions selected from a plurality of predetermined emotions, and wherein each emotion of the plurality of predetermined emotions is associated with modification data indicating how the emotion affects the visual output.

13. The in-vehicle immersive display system of claim 12, wherein the driver emotion profile is a regression output represented by one or more tensors.

14. The in-vehicle immersive display system of claim 10, wherein the display output and the light output are determined based on the use of a color transfer algorithm, wherein the color transfer algorithm generates output color information based on input color information, and wherein the input color information is based on color information within the image data, or is generated based on the color information.

15. The in-vehicle immersive display system of claim 14, wherein the image data represents an image of a scene, and wherein the color information within the image data corresponds to color information of one or more pixels within a determined area of ​​the image. 16 . The in-vehicle immersive display system of claim 15 , wherein the determined region of the image is determined based on an output of a classifier, and wherein the classifier is used to classify the portion of the image. 17 . The in-vehicle immersive display system of claim 15 , wherein the determined area of ​​the image is predetermined based on one or more pixel coordinates of the image.

18. An in-vehicle immersive display system according to claim 10, wherein the controller is further configured to continuously obtain updated image data from the outward-facing camera, wherein a visual output is displayed by the immersive display system and is composed of the display output and the light output, wherein the determination operations are each performed using the updated image data to determine an updated visual output as the visual output for each iteration, and wherein the display of the visual output is continuously updated based on the updated visual output.

19. The in-vehicle immersive display system of claim 10, wherein the visual output is displayed by the immersive display system and consists of the display output and the light output, and wherein the visual output is information indicating an output for a display screen and an output for an ambient light source.

20. An in-vehicle immersive display system, the in-vehicle immersive display system comprising: an electronic display mounted in the vehicle and configured to provide a display output; a diffuse light generating device mounted in the vehicle and configured to provide a light output; an exterior-facing camera mounted in the vehicle and configured to provide image data; a driver monitoring sensor installed in the vehicle and configured to provide driver monitoring sensor data; and A controller is mounted in the vehicle and is configured to determine the display output and the light output based at least in part on the image data provided by the exterior-facing camera and the driver monitoring sensor data provided by the driver monitoring sensor.

Images