Obtaining light distribution information of a physical space

Abstract

A system and method for obtaining light distribution information for a physical space, the system may include a camera for capturing a video feed of a physical space. The system may include a 3D model of the physical space and a Bidirectional Reflectance Distribution Function (BRDF) look-up table. The system may include an electronic processor configured to: map the video feed to the 3D model to form a real-time 3D model of the physical space, extract light properties associated with materials, apertures, and emissive light sources in the physical space from the real-time 3D model using the BRDF look-up table. The system may include a ray tracing algorithm for identifying a location of a light source, determining an angle of reflection from the light source on a surface within the physical space, determining a viewer position within the physical space, and determining a light characteristic at the viewer position from the light source.

Description

D24026W001OBTAINING LIGHT DISTRIBUTION INFORMATION OF A PHYSICAL SPACEBACKGROUND1. Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 700,838, filed on 30 September 2024, and European Application No. 25154571.1 filed on 28 January 2025, each of which is incorporated by reference herein in its entirety.2. Field of the Disclosure

[0002] This application relates generally to systems and methods for obtaining light distribution information in a physical space.3. Description of Related Art

[0003] Automatic or manual brightness level compensation technology is available in current display technology. Actively changing the brightness level automatically by active brightness compensation algorithms or by manual user input is one way to increase the quality for a display when ambient light changes the quality of material being consumed or read on the display. Passive ambient compensation technologies have also been implemented, such as in the television space with anti-glare films. In an environment where the ambient light is constantly changing, such as in the automotive sector, illumination sensors are employed to compensate for the change in ambient light as a vehicle moves.BRIEF SUMMARY OF THE DISCLOSURE

[0004] Examples described herein provide for utilizing a camera real-time video feed associated with a physical space, such as a vehicle interior, in combination with a pre -built 3D model of the car interior enables prediction of the direct and diffuse ambient light impact on any given point in the physical space.

[0005] The 3D model of the vehicle interior provides a realistic representation of the vehicle interior, including physical surface properties. In order to decrease a need for multiple sensors, vehicle manufacturers are transitioning away from designated sensors and are instead utilizing interior cameras to determine the number of or location of passengers within the vehicle. UtilizingD24026W001 the existing interior camera for producing a livestream of the vehicle interior enables a real-time rebuilding of the 3D model to represent a current environmental state of the vehicle interior. By rebuilding the 3D model, information regarding real-time diffuse and direct ambient light sources may be collected.

[0006] Materials used to form and build a vehicle interior typically do not change or move significantly over the lifespan of a vehicle. Accordingly, a 3D model with surface properties and a distribution function may be used to simulate and predict, how light propagates within the vehicle in response to the materials and their associated reflectance, transmission, and diffraction properties as well as texture information. Surface normal data may also be used to develop a ray diagram to help predict the light behavior. This includes for example, how the light will affect displays in the vehicle interior in terms of an amount of glare or black- level lift impacts the output quality of the display.

[0007] Predictions regarding the direct and diffuse ambient light impacts may be used to change and adjust the ambient light in the vehicle interior, allowing for a more enjoyable passenger experience as well as a safer driving environment.

[0008] Various aspects of the present disclosure relate to a light propagation system including: a camera for capturing a video feed of a physical space; at least one light manipulating device; and a controller including an electronic processor and a memory, the memory storing a 3D model of the physical space and a light distribution function, the controller configured to: map the video feed to the 3D model to form a real-time 3D model of the physical space; extract light properties associated with materials in the physical space from the real-time 3D model using the light distribution function; identify a location of a light source; determine an angle of reflection on a surface from the light source based on the light properties associated with the surface within the physical space; determine a viewer position within the physical space; determine a light characteristic at the viewer position from the light source; and send a command to the at least one light manipulating device based on the light intensity.

[0009] In various other aspects the present disclosure relates to a method of obtaining light distribution information for a physical space, the method including: mapping, with an electronic processor, a video feed to a 3D model to form a real-time 3D model of the physical space; identifying, with a ray tracing algorithm, a location of a light source; determining, with the ray tracing algorithm, an angle of reflection from the light source on a surface within the physical space;D24026W001 determining, with the ray tracing algorithm, a viewer position within the physical space; determining, with the ray tracing algorithm, a light characteristic at the viewer position from the light source; and sending a command from the electronic processor to at least one light manipulating device in the physical space based on the light characteristic.

[0010] In some other various aspects, the disclosure herein relates to a method of obtaining light distribution information for a physical space, the method including: retrieving, with an electronic processor, a 3D model of the physical space; identifying, with a ray tracing algorithm, a location of a light source in the physical space; controlling, with a controller, at least one light manipulating device in the physical space based on the location of the light source.

[0011] In this manner, various aspects of the present disclosure provide for capturing light distribution information associated with the physical space, and effect improvements in at least the technical fields associated with display quality in an environment where light distribution is changing such as the transportation sector.DESCRIPTION OF THE DRAWINGS

[0012] These and other more detailed and specific features of various embodiments are more fully disclosed in the following description, reference being had to the accompanying drawings, in which:

[0013] FIG. 1 illustrates a vehicle interior with a light propagation system according to some aspects of the disclosure herein.

[0014] FIG. 2 illustrates a ray diagram for directional reflectance and surface normal, which is used to define a bidirectional reflectance distribution function (BRDF).

[0015] FIG. 3 illustrates a sample chart of various materials with varying BRDFs.

[0016] FIG. 4 illustrates an exemplary ray diagram for the light propagation system according to some aspects of the disclosure herein.

[0017] FIG. 5 illustrates a camera view of an object from FIG. 4.

[0018] FIG. 6 illustrates a virtual image of the object from FIG. 4.

[0019] FIG. 7 illustrates a more detailed ray diagram for the light propagation system according to some aspects of the disclosure herein.D24026W001

[0020] FIG. 8 illustrates a block diagram representing a control system for the light propagation system of FIG. 1 according to some aspects of the disclosure herein.

[0021] FIG. 9 illustrates a flow chart for a method of obtaining light distribution information for the vehicle interior of FIG. 1 according to some aspects of the disclosure herein.

[0022] FIG. 10 illustrates a simplified flow chart a method of obtaining light distribution information for the vehicle interior of FIG. 1 according to some aspects of the disclosure herein.

[0023] FIG. 11 illustrates another simplified flow chart for a method of obtaining light distribution information for a physical space according to some aspects of the disclosure herein.DETAILED DESCRIPTION

[0024] This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. The foregoing is intended solely to give a general idea of various aspects of the present disclosure, and does not limit the scope of the disclosure in any way.

[0025] In the following description, numerous details are set forth, such as device configurations, timings, operations, and the like, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely exemplary and not intended to limit the scope of this application.

[0026] Moreover, while the present disclosure focuses mainly on examples in which the various embodiments are used in a vehicle, it will be understood that this is merely one example of an implementation. It will further be understood that the disclosed systems and methods can be used in any environment in which there is a need to identify light distribution from direct and diffuse ambient light in a physical space.

[0027] The amount of ambient light in an environment affects how a user consumes and reads content on a display screen. When the ambient light changes, the quality of output for the display may decrease.D24026W001

[0028] In an environment where the ambient light is constantly changing, such as in the automotive sector, known ambient compensation technologies become less effective. Currently, illumination sensors are employed to compensate for the change in ambient light as a vehicle moves. These compensation technologies may lag, for example when entering or exiting a tunnel the display may be difficult to read as the light change is quick rather than gradual. Dark and black tones may be significantly impacted in terms of loss of detail. Further, vehicle interiors reflect ambient light that can either impact a display directly or cause glare in the eye of a viewer looking at the display. Identifying direct and diffuse ambient light distribution in a vehicle is complex as there are typically more uncontrollable variables present due to the vehicle moving and changing direction. Also, the outside illumination sources, properties and spatial and temporal behavior can vary profoundly and quickly.

[0029] Identifying all the direct and diffuse ambient illumination properties in a vehicle with individual sensors is challenging. Additional sensors may provide a solution for accessing more input in terms of light distribution. However, adding sensors increases cost and technology within the vehicle which may make the vehicle economically and technically undesirable. Examples described herein provide for utilizing a single camera to determine how many passengers are in a vehicle from the associated video feed rather than with dedicated sensors built into each seat. In this manner, the number of components installed within the vehicle may be reduced.

[0030] Utilizing existing components for enhancing a vehicle environment cuts cost and increases the customer experience. As will be discussed herein, an existing camera is utilized for capturing the current environmental state of a vehicle interior. The existing camera may be a high dynamic range (HDR) camera. Information gathered by the existing camera includes video image pixels that represent the information about real-time diffuse and direct ambient light sources. Diffuse and directional ambient light distribution, such as light reflected off from materials within the vehicle interior can range from directional (e.g. glossy surfaces) to Lambertian (matte surfaces) and may produce glare on, or change the quality of output for, a display in the vehicle interior. Further, directly emitted ambient light from inside or outside the vehicle may also change the quality of the display output if direct light is on and contributing to the ambient light (an interior dome light) or present outside the vehicle (an exterior streetlight). While the existing camera captures a current environment, light emissions or reflections that are visible to the camera may not necessarily be visible to a driver or a passenger in the vehicle and vice versa. Therefore, determining the effects ofD24026W001 diffuse and direct ambient light in other locations of the vehicle interior is beneficial to enhancing the customer experience. Determining these effects can be carried out using indirect methods as described herein.

[0031] As will be discussed in more detail, the materials in a vehicle typically do not change or move significantly over the lifespan of a vehicle. Therefore, material properties, such as texture, reflectance, and directionality can be assumed to be static. This enables building of a 3D model of the vehicle interior, which provides a foundation for determining the diffuse and direct ambient light distribution within the vehicle interior.

[0032] FIG. 1 shows an exemplary vehicle interior 100 of a vehicle 112. The vehicle interior 100 includes multiple material surfaces 114. The material surfaces 114 range in terms of their associated reflectance properties (textures, emission, transmission, etc.). A first material surface 114a may have a highly directional reflectance, for example a metallic seat buckle holder. A second material surface 114b may have a partially diffuse reflectance, for example a leather head rest. A third material surface 114c may have a fully diffuse or Lambertian reflectance, for example a matte finish door. The vehicle interior 100 may be any vehicle interior including that of a traditional vehicle, a hybrid electric vehicle, an electric vehicle, a sports utility vehicle, a truck, a van, a bus, etc.

[0033] The exemplary vehicle interior 100 may include at least one image display unit, referred to herein as a display 116. By way of example the vehicle interior 100 includes a first display 116a, a second display 116b, and a third display 116c. The first display 116a may be in place of or integrated with a rearview display, e.g., a rearview mirror or a rearview electronic display unit. The second display 116b may provide a graphical user interface (GUI) serving as a car infotainment system. The third display 116c may be a rear seat entertainment display screen. While three displays 116 are illustrated, any number of displays are contemplated, including displays that may be removable from the vehicle interior 100, e.g., smartphones, tablets, or gaming consoles.

[0034] The exemplary vehicle interior 100 may further include emissive light sources 118, for example a dome light for illuminating the interior. Other light sources are also contemplated, e.g., LED strips or an array of LEDs providing areas in white or color. The vehicle 112 may also include transparent or partially transparent / reflective, e.g. semi-mirrored or tinted, openings 128 such as windows or moon-roofs, that contribute to the ambient light in the vehicle 112. The light contribution, size and location of an external light source 410 (FIG. 4) shining through theD24026W001 transparent openings 128 in the vehicle 112 can be identified by computing the origin properties (e.g. direction) of the light source based on the reflectance properties of the surface material, the light is reflected from.

[0035] A light propagation system 120 may be mounted within or integral with the exemplary vehicle interior 100. The light propagation system 120 includes at least one sensor, by way of example an interior camera 122i. The interior camera 122i may point from a rear 123 of the vehicle interior 100 towards a front 124 of the vehicle interior 100. The interior camera 122i may be mounted near or integrated with any portion of the rear 123 of the vehicle. While illustrated as pointing from the rear 123 towards the front 124, the interior camera 122i may be located anywhere in the vehicle, including at the front 124 or integral with any of the displays 116. The interior camera 122i is arranged to provide a video feed 126, represented by a dashed line, of the vehicle interior 100. The video feed 126 may include real-time video footage, still images captured at timed intervals, depth information, or other directional information, or any combination thereof.

[0036] While one interior camera 122i is illustrated, two or more interior cameras are contemplated. Additional cameras may help improve accuracy as the light contributions can be compared at differing angles. Further, an exterior camera 122e for the vehicle 112 may contribute to the video feed 126. The feed from the exterior camera 122e may be used to identify the origin of external light sources.

[0037] The light propagation system 120 may include a pre-built 3D model 130 of the vehicle interior 100, referred to herein simply as 3D model 130. The 3D model 130 may be stored in a memory 806 (see FIG. 8). Since the material surfaces 114 are static, the 3D model 130 of the vehicle interior 100 provides a base reference for the light propagation system 120. The 3D model 130 may be in the form of a polygon mesh where the polygons arc textured according to the different reflectance properties of the material surfaces 114. The light sources 118 and the transparent openings 128 may be identified in the 3D model 130. The transparent openings 128 can be treated as apertures for light incident on any material surfaces 114, in this manner, a potential angle range a material can be illuminated is limited.

[0038] The light propagation system 120 may include a control system 140. The control system 140 may be configured to receive communication from any of the displays 116 and the cameras 122i,D24026W001122e. Further the control system 140 may have access to the 3D model 130. The 3D model 130 may also be stored in a separate server that may be accessed wirelessly by the control system 140.

[0039] A light distribution function, by way of example a bidirectional reflectance distribution function (BRDF) 132, for the material of each reflective surface, as defined by Equation (1), may also be stored in the memory 806. . .. Equation (1)~

[0040] The BRDF is a mathematical function that defines how light from a source is reflected off an opaque surface, as illustrated in FIG. 2. The function takes an incoming light direction, coi, and an outgoing light direction, cor, and returns the ratio of reflected radiance L exiting along the outgoing light direction corto the irradiance E incident on the surface from the incoming light direction coi. The angle 0i is measured between coi and a surface normal n. Rather than simple angles, each direction is modeled using spherical coordinates (q>,0) making the BRDF a four-dimensional function.

[0041] The BRDF describes a material’s albedo (e.g., the fraction of light that a surface reflects). A highly reflective surface where all light is reflected has an albedo equal to 1. The BRDF further describes the reflectance directions for both the incoming and outgoing light directions coi, cor. An example set of spheres (e.g. based on the beforementioned polygons) with varying BRDFs 132 applied, simulating a multitude of materials is illustrated in FIG. 3.

[0042] The BRDF only describes the reflectance properties of a solid, non-transparent material. It is further contemplated that more complex properties such as transmissivity, sub-surface scatter, anisotropy, participating media, etc. may be considered in determining the light scattering behavior. These properties can be described by a bidirectional scattering- surface reflectance distribution function or BSSRDF. This extension of the BRDF also includes subsurface scattering properties, including transparencies. This can be applicable to more complex materials in a car such as transparent plastics, glass or crystal applications that have become common.

[0043] FIG. 4 illustrates an exemplary ray diagram 400 utilized by a ray tracing algorithm 420 for the light propagation system 120. The ray tracing algorithm 420 may also be stored in the memory 806 (FIG. 8). A camera 422 is illustrated in a fixed / known position. As the camera 422 may be an existing camera, e.g. the interior camera 122i for the vehicle 112, the fixed / known position of theD24026W001 camera 422 may be indicated by the 3D model 130. For a given light source 410, light travels from the light source 410 to the surface of an object 412 and to the camera 422 along a first ray path 414. The reflection behavior on the surface of the object 412 is given by the 3D model 130 and the BRDF 132. Utilizing a ray tracing algorithm 420, additional ray paths may be modeled and the light distribution, including points of reflection from the light source 410, points of transmission for the light source 410, and / or points of origin of the light source 410 within or outside of the vehicle interior 100 may be computed. The ray tracing algorithm 420 may include, but is not limited to, the following steps: computing the location of light sources within and outside the vehicle 112, determining an angle of reflection from the light source on a surface within the vehicle interior 100, and determining a light characteristic for various viewer locations, e.g., the viewer position 416, within the vehicle interior 100. The light characteristic may include intensity, color, spectral composition or the like.

[0044] The camera 422 captures a camera viewpoint 510 of the object 412 for example a material or object within the vehicle interior 100. The camera viewpoint 510 is illustrated in FIG. 5. The camera viewpoint 510 includes a first light characteristic, in one example, a first amount of reflected light intensity 514. Based on known attributes of the object 412, the shape being a circular cylinder and the BRDF being that of a silver anisotropic metal, a location of the light source 410 may be computed.

[0045] Turning to FIG. 6, a second amount of reflected light intensity 612 and a third amount of reflected light intensity 614 as would be seen from other locations within the vehicle 112 may be determined based on the computer location of the light source 410. A virtual image 610 represents a second viewpoint and may be computed based on the viewer position 416. In other words, the virtual image 610 is a rendering of what a viewer would see when light travels from the light source 410 along a second ray path 418 (FIG. 4) to the viewer position 416. The second and third amounts of reflected light intensities 612, 614 are illustrated in the virtual image 610 when compared to the first amount of reflected light intensity 514 of the camera viewpoint 510. Knowing the light distribution for the viewer position 416 provides information for possible adjustments, whether automatic or manual, resulting in an improved display quality output.

[0046] FIG. 7 is a more detailed ray diagram 700 for aiding in determining the light distribution within the vehicle interior 100 with the light propagation system 120. A light source 710 shines light onto an object, for example a dashboard 712 of the vehicle 112. A light ray 714 incident on theD24026W001 surface of the dashboard 712 at point is reflected. If the surface of the dashboard 712 is a perfect reflector (an approximation would be a mirror surface), the angle of incidence a is equal to the angle of reflection y. Typically, surfaces are not perfect reflectors and therefore, when reflected, the light ray 714 is scattered to some extent. The shaded area 716 provides an example of such scattering behavior. The angle of reflection y, provides a location along which the highest, or peak intensity of light is reflected, 100%. Angles beyond the angle of reflection y, marked as 75%, 50%, and 25%, indicate reflected light having less and less intensity. As previously discussed herein, the reflectance properties of a surface of the dashboard 712 are known and present in the 3D model 130. The behavior of this decrease in intensity is described by the BRDF for a given material.

[0047] When computing the light distribution in the vehicle interior 100, it is important to also consider the inverse square law where an intensity I of light decreases with the square of the distance d from the light source, as provided by Equation (2): Equation (2)

[0048] A camera 722, e.g., the interior camera 122i includes individual pixels, where each pixel captures an amount of light characteristic. An offset angle 0 for each individual pixel is determined utilizing the ray tracing algorithm 420. The offset angle 0 is measured to the surface normal n with respect to dashboard 712. Knowing the offset angle 9, and in combination with the BRDF of the material at point A, an amount of light characteristic for each pixel may be determined. In this example the intensity I of light at the camera 722 is 88% of the peak intensity.

[0049] Knowing the offset angle 0 and the amount of light intensity enables the light propagation system 120 to determine an amount of light intensity reflected to potential viewers. For example, a viewer position 718 of a driver, may have the same angle as the angle of reflection y and therefore receive 100% of the peak intensity whereas a viewer position 720 for a rear seat passenger, with angle 5, may only receive 72% of the peak intensity.

[0050] When computing and solving for the origin of light sources for a camera with a large number of pixels, there will likely be several pixels, e.g., a line of pixels, or an area of pixels, e.g., an array of pixels, that receive light that originates from the same illumination source. This information can be used to increase robustness for the ray tracing algorithm 420 in determining the reflectanceD24026W001 properties for other locations of the vehicle interior 100 if the surface is less directionally reflecting.In other words, more pixels may equate with a more accurate virtual image 610.

[0051] The light propagation system 120 may also identify camera pixels representing any emissive light sources 118 in the field of view. Further, a feedback loop may be implemented that communicates if any emissive light source 118 is currently switched on or off.

[0052] The computations described herein are most accurate for point-like light sources when the material has a significant directional component, e.g., glossy, or semi-gloss. Common to the automotive area, examples of such materials include polished wood or aluminum panels, knobs, or glass applications. These types of surfaces are more likely to cause undesirable localized effects. For example, the sun or headlights of other vehicles in the night can create specular reflections causing glare or spatially complex reflections on instruments and displays.

[0053] With more Lambertian reflection properties, it is likely that multiple light sources, or area light sources such as the sky can contribute to the intensity level captured by each camera pixel. This may cause less accuracy when assessing the direction of the light source, however as the reflecting material is diffusing and dispersing the light into a wide-angle area, the area of the material captured by the camera pixel will contribute a similar amount of light to other areas in the vehicle interior and a directional vector identifying the origin of the light source is therefore not as significant.

[0054] As clothes and other kind of fabrics are typically more diffusely reflecting, passengers in the vehicle 112 do not need to be represented by the 3D model 130. The ray tracing algorithm 420 may assume the reflectance of passenger areas, when occupied, to be Lambertian. It is further contemplated that a real-time depth map may be generated by the control system 140 based on the video feed 126, such that simple reflections may also be assessed, e.g., ones occurring on skin or glasses.

[0055] FIG. 8 provides a block diagram 800 representing the control system 140 of the light propagation system 120. All or parts of the control system 140 may be integral with the second display 116b, including a controller 802. The controller 802 includes, among other things, an electronic processor 804, a memory 806, and a transceiver 814. The electronic processor 804, the memory 806, and the transceiver 814 communicate over one or more control and / or data buses. FIG. 4 illustrates only one example of the controller 802. The controller 802 may include more or fewerD24026W001 components and may perform functions other than those explicitly described herein. In some examples, the electronic processor 804 is implemented as a microprocessor with separate memory 806. In other examples, the electronic processor 804 is implemented as a microcontroller, where the memory 806 is on the same chip. The electronic processor 804 may be implemented with multiple processors, and may be implemented partially or entirely as, for example, a field-programmable gate array (FPGA) or an applications specific integrated circuit (ASIC).

[0056] The memory 806 includes non-transitory, computer-readable memory or medium that stores instructions that are received and executed by the electronic processor 804 to carry out the functionality of the light propagation system 120 described herein. The memory 806 may include, for example, combinations of different types of memory, such as read-only memory and randomaccess memory. The 3D model 130 including color, texture, transparency, and the BRDF 132, may be stored in the memory 806. A position of the cameras 122i, 122e, 422, 722 along with positions associated with significant elements, e.g., displays 116, in the vehicle 112 may also be stored in the memory 806.

[0057] The transceiver 814 allows the controller 802 to perform wired and / or wireless communications with any of the displays 116. The communication may occur over a network 820. The transceiver 814 may also handle communication with various components and devices of the light propagation system 120 connected to the controller 802, such as the camera 122.

[0058] The controller 802 receives and processes data from connected components to operate the light propagation system 120. For example, the controller 802 receives the video feed 126 from the interior camera 122i of vehicle interior 100 and receives the 3D model 130 of the vehicle interior 100 from the memory 806. The video feed 126 and the 3D model 130 are processed together to deduct and predict how the ambient light is distributed within the vehicle interior 100 at any given location. It is also possible to create a real-time spherical Image Based Lighting (IBL) map with exterior cameras that can be processed as well. Such an IBL map can aid in identifying direction and properties of light sources external to the vehicle.

[0059] A flow chart for a method 900 of obtaining light distribution information for the vehicle interior 100 described herein is illustrated in FIG. 9. The method 900 may be performed by, for example, the controller 802. Additionally, the steps provided within FIG. 9 are merely examples,D24026W001 and may instead be conducted in a different order or simultaneously. The method 900 may be performed in the vehicle 112 using the light propagation system 120, described herein.

[0060] At block 910, the method 900 includes accessing stored information including the 3D model 130, the BRDF 132, the position of the cameras 122i, 422, 722 in the vehicle 112, the position of any exterior cameras 122e, and the position of significant elements such as the displays 116. The stored information may be fed to the ray tracing algorithm 420 described herein.

[0061] At block 912, the method 900 includes capturing the virtual image 610 from the vantage point of the camera 122.

[0062] At block 914, the method 900 includes feeding the video feed 126, a two-dimensional livestream of the vehicle interior 100, to the controller 802.

[0063] At block 916, the method 900 includes mapping the input from the video feed 126 to the 3D model 130 to define a real-time 3D model of the vehicle interior in order to align the pixel intensities captured by the camera with the corresponding BRDF and surface properties of the 3D model. In one example, at the time of manufacturing, some level of pre-alignment may be performed such that a portion of the 3D model 130 already includes interior components of the vehicle 112. Then, during operation, a real-time alignment may occur based on key features in the frame, by way of example, transitions between materials, easily identifiable objects such as knobs, buttons with text, etc. to complete the real-time 3D model.

[0064] At block 918, the method 900 includes determining the offset angle 9 for each individual pixel of the camera 122 from the real-time 3D model.

[0065] At block 920, the method 900 includes determining occlusions such as passengers and objects in the vehicle by excluding information about the material in these areas from the ray tracing algorithm 420. In other words, excluding information about materials in occluded areas of the vehicle interior 100 from the ray tracing algorithm 420 as the camera pixels representing these occluded areas are not represented in the materials described by the 3D model.

[0066] At block 922, the method 900 includes determining the light source location. As previously discussed herein, calculating the light source location from multiple pixels increases accuracy in determining the light source position as well as the intensity of the light source(s). Diffuse reflections may be utilized to deduct diffuse light from the real-time 3D model. In some cases,D24026W001 utilizing an HDR camera, having a large dynamic range and signal to noise ratio, can help with accuracy.

[0067] At block 924, the method 900 includes computing the viewer position 416, 718, 720 from the video feed 126 by utilizing eye or head tracking.

[0068] At block 926, the method 900 includes extracting emission / reflectance properties of materials, apertures, and light sources in the vehicle interior 100 from the real-time 3D model using the BRDF 132 These inputs along with the viewer position 416, 718, 720 enable producing of the virtual image 610.

[0069] At block 928, the method 900 includes determining a light intensity from the virtual image 610 and identifying light sources that change a quality of output for the displays 116, for example by producing glare for one or more viewers and the potential impact the glare may have on the displays 116. Changing the quality of output for the displays may also include producing a black level increase on the displays 116.

[0070] At block 930, the method 900 includes providing the resulting virtual image 610 and the associated intensities to a spatio-temporal ambient compensation algorithm (or other ambient light compensation algorithms).

[0071] A simplified flow chart for a method 1000 of obtaining light distribution information for the vehicle interior 100 described herein is illustrated in FIG. 10. The method 1000 may be performed by, for example, the controller 802. Additionally, the steps provided within FIG. 10 are merely examples, and may instead be conducted in a different order or simultaneously. The method 1000 may be performed in the vehicle 112 using the light propagation system 120, described herein.

[0072] At block 1010, the method 1000 includes mapping, with the electronic processor 804, the video feed 126 to the 3D model to form a real-time 3D model of the physical space. The real-time 3D model may include, for example, light color and intensity information from the video feed and geometry and material properties from the stored 3D model.

[0073] At block 1020, the method 1000 includes identifying, with a ray tracing algorithm 420, a location of a light source.D24026W001

[0074] At block 1030, the method 1000 includes determining, with the ray tracing algorithm 420, an angle of reflection from the light source on a surface within the physical space.

[0075] At block 1040, the method 1000 includes determining, with the ray tracing algorithm 420, a viewer position within the physical space.

[0076] At block 1050, the method 1000 includes determining, with the ray tracing algorithm 420, a light intensity at the viewer position from the light source.

[0077] At block 1060, the method 1000 includes sending a command from the electronic processor to the at least one light manipulating device based on the light intensity. In one example, sending a command may include sending a signal to the display(s) 116 to lift black level. In another example the command may be to adjust interior lighting, such as LEDs. In even another example, the command could adjust mechanical, photoelectrical or photochemical light barriers such as blinds. Adjustments may include complementing the determined light distribution and homogenize the light environment within the vehicle. Adjustments may include changing diffuse light patterns within the vehicle to dampen the effect of the glare(s) identified. These changes may be steady or continuous depending on environment through which the vehicle is traveling. For example, in a tunnel where lighting in the tunnel may produce a spatial and / or temporal flashing effect within the vehicle, the interior lighting may respond with a spatially and / or temporally opposing flashing to compensate, dampen or eliminate potential flashing effects on the displays 116 or other surfaces. These adjustments may be temporally dampened to prevent unintentional jumps in light compensation.

[0078] Further, the vehicle interior 100 may be fitted with actively changing materials that can be mechanical, photochemical and photoelectrical. For example, the windows may be made from material capable of attenuating an amount of light coming into the vehicle by changing the albedo and / or transmissivity of the glass. These same types of materials may be used to change the car’s interior color, e.g., for normal driving conditions a light color and for content consumption a dark color, again by using mechanical, photochemical, photoelectrical materials such as electrophoretic surfaces.

[0079] Another simplified flow chart for a method 1100 of obtaining light distribution information for a physical space, e.g., the vehicle interior 100 described herein, is illustrated in FIG. 11. The method 1100 may be performed by, for example, the controller 802. Additionally, the steps provided within FIG. 11 are merely examples, and may instead be conducted in a different order orD24026W001 simultaneously. The method 1100 may be performed in a physical space, e.g., the vehicle 112, using the light propagation system 120, described herein.

[0080] At block 1110, the method 1100 includes retrieving, with the electronic processor 804, the 3D model 130 of a physical space. While described herein as a vehicle interior 100, it should be understood that the physical space may be any defined physical space including a building room, an aircraft cabin, a boat cabin, a film set, or the like.

[0081] At block 1120, the method 1100 includes identifying, with the ray tracing algorithm 420, a location of a light source in the physical space. The light source may be the sun, a streetlight, a dome light, a display, or the like.

[0082] At block 1130, the method 1100 includes controlling, with the controller 802, at least one light manipulating device, e.g., the second display 116b, in the physical space based on the location of the light source.

[0083] While described with regards to a vehicle interior, it should be understood that the concepts discussed herein may be implemented outside of the automotive sector. In any environment where the geometric setup does not significantly change, e.g., in developing architectural concepts, the light propagation system described herein may be used to identify and compensate for complex ambient light situations.

[0084] Systems, methods, and devices in accordance with the present disclosure may take any one or more of the following configurations.

[0085] Clause 1. A light propagation system comprising: a camera for capturing a video feed of a physical space; at least one light manipulating device; and a controller including an electronic processor and a memory, the memory storing a 3D model of the physical space and a light distribution function, the controller configured to: map the video feed to the 3D model to form a real-time 3D model of the physical space; extract light properties associated with materials in the physical space from the real-time 3D model using the light distribution function; identify a location of a light source; determine an angle of reflection on a surface from the light source based on the light properties associated with the surface within the physical space; determine an angle of reflection from the light source on a surface within the physical space; determine a viewer position within the physical space; determine a light characteristic at the viewer position from the lightD24026W001 source; and sending a command to the at least one light manipulating device based on the light characteristic.

[0086] Clause 2. The light propagation system of clause 1, wherein the at least one light manipulating device is a display located within the physical space and wherein the electronic processor is configured to identify light sources that change the quality of output on the display.

[0087] Clause 3. The light propagation system of clause 2, wherein the quality of output is affected by light sources that produce glare on the display.

[0088] Clause 4. The light propagation system of any one of clauses 2-3, wherein the quality of output is affected by high diffuse ambient light that produces a black level increase on the display.

[0089] Clause 5. The light propagation system of any one of clauses 1-4, further comprising extracting light properties associated with apertures in the physical space.

[0090] Clause 6. The light propagation system of any one of clauses 1-5, further comprising extracting light properties associated with emissive or reflective light sources in the physical space.

[0091] Clause 7. The light propagation system of any one of clauses 1-6, wherein the light properties include emissive and reflectance properties.

[0092] Clause 8. The light propagation system of any one of clauses 1-7, wherein the physical space is a vehicle interior.

[0093] Clause 9. The light propagation system of any one of clauses 1-8, wherein the electronic processor is configured to determine an offset angle 0 with respect to the angle of reflection for each individual pixel of the camera.

[0094] Clause 10. The light propagation system of any one of clauses 1-9, wherein the electronic processor is configured to determine an offset angle 0 with respect to the angle of reflection for an array of pixels of the camera.

[0095] Clause 11. The light propagation system of any one of clauses 1-10, wherein the electronic processor is configured to determine an offset angle 0 with respect to the angle of reflection for a line of pixels of the camera.D24026W001

[0096] Clause 12. The light propagation system of any one of clauses 1-11, wherein the electronic processor is configured to produce a virtual image at the viewer position and to identify light sources that produce glare from the virtual image.

[0097] Clause 13. The light propagation system of any one of clauses 1-12, further comprising an additional camera for providing an additional video feed of the physical space at a different angle for further mapping to the 3D model.

[0098] Clause 14. The light propagation system of any one of clauses 1-13, further comprising an exterior camera for providing image data of an exterior of the physical space to the controller to locate exterior light sources.

[0099] Clause 15. The light propagation system of any one of clauses 1 -1 , wherein the light distribution function is a Bidirectional Reflectance Distribution Function (BRDF).

[0100] Clause 16. The light propagation system of any one of clauses 1-14, wherein the light distribution function is Bidirectional Scattering-Surface Reflectance Distribution Function (BSSRDF).

[0101] Clause 17. A method of obtaining light distribution information for a physical space, the method comprising: mapping, with an electronic processor, a video feed to a 3D model to form a real-time 3D model of the physical space; identifying, with a ray tracing algorithm, a location of a light source; determining, with the ray tracing algorithm, an angle of reflection from the light source on a surface within the physical space; determining, with the ray tracing algorithm, a viewer position within the physical space; determining, with the ray tracing algorithm, a light characteristic at the viewer position from the light source; and sending a command from the electronic processor to at least one light manipulating device in the physical space based on the light characteristic.

[0102] Clause 18. The method of clause 17, further comprising producing the video feed of the physical space with a camera within the physical space.

[0103] Clause 19. The method of clause 18, further feeding stored information, including the 3D model, a position of the camera, and a position of significant elements in the physical space to the ray tracing algorithm.D24026W001

[0104] Clause 20. The method of any one of clauses 18-19, further comprising determining an offset angle from the angle of reflection for each individual pixel of the camera.

[0105] Clause 21. The method of any one of clauses 17-20, further comprising extracting, with the electronic processor, light properties associated with materials, apertures, and emissive light sources in the physical space from the real-time 3D model using a distribution function.

[0106] Clause 22. The method of any one of clauses 17-21, further comprising pre-aligning the physical space with the video feed.

[0107] Clause 23. The method of any one of clauses 17-22, further comprising excluding information about material in occluded areas of the physical space from the ray tracing algorithm.

[0108] Clause 24. The method of any one of clauses 17-23, further comprising producing a virtual image at the viewer position.

[0109] Clause 25. The method of clause 24, further comprising identifying light sources that produce glare from the virtual image.

[0110] Clause 26. The method of any of clauses 24-25, further comprising providing the virtual image to a spatio-temporal ambient compensation algorithm.

[0111] Clause 27. A method of obtaining light distribution information for a physical space, the method comprising: retrieving, with an electronic processor, a 3D model of the physical space; identifying, with a ray tracing algorithm, a location of a light source in the physical space; controlling, with a controller, at least one light manipulating device in the physical space based on the location of the light source.

[0112] Clause 28. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of clauses 17-27.

[0113] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described stepsD24026W001 performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claims.

[0114] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0115] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary in made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0116] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

D24026W001CLAIMSWhat is claimed is:1 . A method of obtaining light distribution information for a physical space, the method comprising: mapping, with an electronic processor, a video feed to a 3D model to form a real-time 3D model of the physical space; identifying, with a ray tracing algorithm, a location of a light source; determining, with the ray tracing algorithm, an angle of reflection from the light source on a surface within the physical space; determining a viewer position within the physical space; determining, with the ray tracing algorithm, a light characteristic at the viewer position from the light source based on the angle of reflection; and sending a command from the electronic processor to at least one light manipulating device in the physical space based on the light characteristic.

2. The method of claim 1, further comprising producing the video feed of the physical space with a camera within the physical space.

3. The method of claim 2, further comprising feeding stored information, including the 3D model, a position of the camera, and a position of significant elements in the physical space to the ray tracing algorithm.

4. The method of claim 2 or 3, further comprising determining an offset angle from the angle of reflection for each individual pixel of the camera.

5. The method of any preceding claim, further comprising extracting, with the electronic processor, light properties associated with materials, apertures, and emissive light sources in the physical space from the real-time 3D model using a distribution function.D24026W0016. The method of any preceding claim, further comprising pre-aligning the physical space with the video feed.

7. The method of any preceding claim, further comprising excluding information about material in occluded areas of the physical space from the ray tracing algorithm.

8. The method of any preceding claim, further comprising producing a virtual image at the viewer position.

9. The method of claim 8, further comprising identifying light sources that produce glare from the virtual image.

10. The method of claim 8 or 9, further comprising providing the virtual image to a spatiotemporal ambient compensation algorithm.

11. A method of obtaining light distribution information for a physical space, the method comprising: retrieving, with an electronic processor, a 3D model of the physical space; identifying, with a ray tracing algorithm, a location of a light source in the physical space; and controlling, with a controller, at least one light manipulating device in the physical space based on the location of the light source.

12. A light propagation system comprising: a camera for capturing a video feed of a physical space; at least one light manipulating device; and a controller including an electronic processor and a memory, the memory storing a 3D model of the physical space and a light distribution function, the controller configured to: map the video feed to the 3D model to form a real-time 3D model of the physical space; extract light properties associated with materials in the physical space from the real-D24026W001 time 3D model using the light distribution function; identify a location of a light source; determine an angle of reflection on a surface from the light source based on the light properties associated with the surface within the physical space; determine a viewer position within the physical space; determine a light characteristic at the viewer position from the light source based on the angle of reflection; and send a command to the at least one light manipulating device based on the light characteristic.

13. The light propagation system of claim 13, wherein the at least one light manipulating device is a display located within the physical space and wherein the electronic processor is configured to identify light sources that change a quality of output on the display.

14. The light propagation system of claim 13, wherein the quality of output is affected by light sources that produce glare on the display.

15. The light propagation system of claim 13, wherein the quality of output is affected by high diffuse ambient light that produces a black level increase on the display.

16. The light propagation system of any one of claims 12 to 15, further comprising extracting light properties associated with apertures in the physical space.

17. The light propagation system of any one of claims 12 to 15, further comprising extracting light properties associated with emissive or reflective light sources in the physical space.

18. The light propagation system of claim 17, wherein the light properties include emissive and reflectance properties.

19. The light propagation system of any one of claims 12 to 18, wherein the physical space is a vehicle interior.D24026W00120. The light propagation system of any one of claims 12 to 19, wherein the electronic processor is configured to determine an offset angle 0 with respect to the angle of reflection for each individual pixel of the camera.

21. The light propagation system of any one of claims 12 to 20, wherein the electronic processor is configured to determine an offset angle 0 with respect to the angle of reflection for an array of pixels of the camera.

22. The light propagation system of any one of claims 12 to 21, wherein the electronic processor is configured to determine an offset angle 0 with respect to the angle of reflection for a line of pixels of the camera.

23. The light propagation system of any one of claims 12 to 22, wherein the electronic processor is configured to produce a virtual image at the viewer position and to identify light sources that produce glare from the virtual image.

24. The light propagation system of any one of claims 12 to 23, further comprising an additional camera for providing an additional video feed of the physical space at a different angle for further mapping to the 3D model.

25. The light propagation system of any one of claims 12 to 24, further comprising an exterior camera for providing image data of an exterior of the physical space to the controller to locate exterior light sources.

26. The light propagation system of any one of claims 12 to 25, wherein the light distribution function is a Bidirectional Reflectance Distribution Function (BRDF).

27. The light propagation system of any one of claims 12 to 25, wherein the light distribution function is a Bidirectional Scattering-Surface Reflectance Distribution Function (BSSRDF).

28. A non-transitory computer-readable medium storing instructions that, when executed by oneD24026W001 or more processors, cause the one or more processors to perform the method of any one of claims 1 to 11.

Images