VEHICLE PROCEDURES FOR ENVIRONMENTALLY FRIENDLY ADAPTATION OF CAMERA DISPLAY
The vehicle system addresses illumination fluctuations by dynamically adjusting camera frame rates and brightness based on ambient light and vehicle state, enhancing visibility and display effectiveness.
Patent Information
- Application Number
- DE102023128706
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-07-13
- Filing Date
- 2023-10-19
- Publication Date
- 2026-01-29
- Estimated Expiration
- 2043-10-19
AI Technical Summary
Existing vehicle camera systems struggle with fluctuations in illumination due to varying lighting conditions, leading to reduced visibility and display effectiveness.
A vehicle system that dynamically adjusts the frame rate and brightness of onboard cameras based on ambient light and vehicle state, using a control module to process image data and adjust display characteristics to maintain optimal visibility.
Enhances visibility by smoothly adapting to changes in lighting and vehicle conditions, improving the driver's perception of the displayed environment.
Smart Images

Figure 00000016_0000 
Figure 00000017_0000 
Figure 00000018_0000
Abstract
Description
INTRODUCTION
[0001] The technical field generally refers to vehicle systems and in particular to the automatic adjustment of the camera view display to improve visibility depending on the current environment.
[0002] Modern vehicles feature various enhanced or advanced functions to assist the driver in operating the vehicle. Many vehicles are capable of perceiving their surroundings and facilitating vehicle operation. For example, many vehicles are equipped with numerous cameras that provide video or images of various areas outside the vehicle, such as rearview cameras, side cameras, surround-view cameras, and / or similar systems. The video or images captured by the onboard cameras are typically displayed on a screen in the dashboard, such as an infotainment display or another screen in the center console. The displayed video or images assist the driver in operating the vehicle, for example, by facilitating parking or other manual vehicle operations where a view of external areas is desired.
[0003] In practice, camera systems and corresponding infotainment displays suffer from fluctuations in illumination due to the sun's azimuth angle (e.g., at dusk and / or sunrise) or shadows cast by other external objects that affect the illumination of the scene outside the vehicle and / or the vehicle interior, such as infrastructure (e.g., overpasses, garages, etc.), buildings, trees, geographical features, and / or similar elements. This leads to perceived fluctuations in display brightness, which in turn reduces the driver's view of the surrounding area(s) and thus the effectiveness of the display. Therefore, it is desirable to provide adaptive camera and display systems for improved visibility and safety, regardless of variations in lighting or the environment.
[0004] US 2014 / 0354811A1 describes a vehicle vision system comprising a camera, a controller, and a display device. The camera is mounted on the vehicle and has a field of view outside the vehicle. The camera has a two-dimensional array of multiple light-sensitive elements. The controller has an image processor capable of processing image data captured by the camera. The display device is visible to the driver of the vehicle and includes a video screen. In response to the detection of a decrease in the light level in the imaged scene, the controller is capable of performing at least one of the following actions: (i) adjusting the exposure time for image capture by the camera, and (ii) adjusting the frame rate for image capture by the camera.The controller is able to adjust the camera to control the display intensity of the video images shown on the video screen.
[0005] EP 3 410 702 A1 describes how the object or environment that is essential to check while driving a vehicle varies depending on the vehicle's operating condition. However, if the brightness or color of the entire viewing angle or a predetermined part of the image is adjusted, the brightness or color of the image may not be appropriate when a driver checks the object or environment that is essential to check while driving.An image acquisition device comprises: an image acquisition unit configured to capture an image of a vehicle's surroundings; a control unit configured to control the image acquisition unit; an image processor configured to process image data output by the image acquisition unit; an output unit configured to output the image processed by the image processor to a display unit; and a detection unit configured to detect information about a change in the vehicle's course. At least one of the image acquisition control by the control unit and the image processing by the image processor applies weighting such that the weighting, based on the course change information detected by the detection unit, increases in the direction of the course change.This image recording device makes it possible to present images that allow the driver to adequately check the condition of the lane after the course change.
[0006] US 2020 / 0096769 A1 describes a near-eye display system. The near-eye display system comprises a display, an image capture device, a storage device, and a processor. The image capture device is configured to capture an image of the field of view in which a user views the display. The storage device is configured to record programs or commands. The processor loads and executes the programs or commands recorded in the storage device to: calculate a brightness distribution in the image field according to the image; set a luminance inversion mask according to the brightness distribution in the image field; and control the display to show according to the luminance inversion mask. Additionally, a display procedure for the near-eye display system is provided. DESCRIPTION
[0007] The object of the invention is to perform a more precise and targeted adjustment of the display brightness according to the different lighting conditions in the identified areas. This object is achieved by the subject matter of claim 1. Further developments are described in the dependent claims.
[0008] Devices for a vehicle and corresponding methods and vehicle systems are provided. One method for supporting the operation of a vehicle involves a control module connected to the vehicle receiving image data acquired by an onboard imaging system using an initial frame rate, receiving current vehicle state information connected to the vehicle, determining a representative exposure setting for the imaging system at least partially based on the current vehicle state information, determining an estimated illuminance of an environment surrounding the vehicle at least partially based on the image data and the representative exposure setting, and automatically configuring the imaging system for a second frame rate, different from the initial frame rate, based on the estimated illuminance.The procedure then receives subsequent image data from the imaging system, which was acquired using the second frame rate, provides a graphical user interface with a graphical representation of the subsequent image data on a display device connected to the vehicle, and adjusts a display characteristic of the graphical user interface over a temporary period of time after the automatic configuration of the imaging system for the second frame rate.
[0009] In some implementations, setting the display characteristic involves incrementally adjusting the brightness of the graphical representation of subsequent image data over a temporary period. In one or more implementations, incrementally adjusting the brightness involves configuring the brightness of the graphical representation of an initial frame of subsequent image data based on a weighted average of the initial frame of subsequent image data captured at the second frame rate and a preceding frame of image data captured at the initial frame rate.In some implementations, the method identifies one or more areas of interest within the image data. Determining the estimated illuminance involves converting RGB component values for the one or more areas of interest to YUV component values and averaging the Y-channel luminance component values of the YUV component values over the one or more areas of interest. In one or more implementations, determining the representative exposure setting involves calculating the representative exposure setting as a weighted sum of the respective exposure settings associated with the respective imaging devices of the imaging system, where a respective weighting factor associated with a particular imaging device is influenced by the current state of a transmission system connected to the vehicle.
[0010] In one or more implementations, the current vehicle state information includes the current state of a vehicle transmission system. Determining the representative exposure setting involves calculating the representative exposure setting as a weighted sum of the respective exposure settings assigned to the respective imaging devices of the imaging system, with a weighting factor assigned to each imaging device being influenced by the current state of the vehicle's transmission system. In one or more embodiments, adjusting the display characteristic involves gradually increasing the brightness of the graphical representation of the subsequent image data over the temporary period following the automatic configuration of the imaging system for a lower frame rate.In other implementations, the adjustment of the display characteristics consists of gradually reducing the brightness of the graphical representation of the subsequent image data over the temporary period following the automatic configuration of the imaging system for a higher frame rate. In still other embodiments, the automatic configuration of the imaging system for the second frame rate involves automatically switching the imaging system from a higher to a lower frame rate when the current state of a vehicle transmission system is reverse gear and a reddening value associated with the image data is greater than a threshold, provided the estimated ambient light level of the vehicle is below a second threshold.In other implementations, adjusting the display characteristics of the graphical user interface involves dynamically adjusting the brightness of the display device, based at least partially on the estimated illuminance.
[0011] A device for a non-transmittable, computer-readable medium is also provided. The non-transitory, computer-readable medium contains executable instructions which, when executed by a processor, cause the processor to provide an adaptive display service that is configurable to obtain image data acquired using an initial frame rate from an onboard vehicle imaging system, to obtain current vehicle state information associated with the vehicle, and to determine a representative exposure setting for the imaging system, at least partially, based on the current vehicle state information.to determine an estimated illuminance of the vehicle's surroundings, at least partially, based on the image data and the representative exposure setting, and to automatically configure the imaging system for a second frame rate based on the estimated illuminance, which differs from the original frame rate. The instructions then instruct the processor to obtain subsequent image data from the imaging system acquired using the second frame rate, to provide a graphical user interface with a graphical representation of the subsequent image data on a display device associated with the vehicle, and to adjust the display characteristics of the graphical user interface for a temporary period after the automatic configuration of the imaging system for the second frame rate.
[0012] In one implementation, the adaptive display service is configurable to incrementally adjust the brightness of the graphical representation of the subsequent image data over a temporary period of time. In another implementation, the adaptive display service is configurable to identify one or more areas of interest within the image data, where determining the estimated illuminance involves converting RGB component values for the one or more areas of interest into YUV component values and averaging the Y-channel luminance component values of the YUV component values across the one or more areas of interest.In another implementation, the adaptive display service is configurable to obtain a current state of a vehicle's transmission system. Determining the representative exposure setting involves calculating the representative exposure setting as a weighted sum of the respective exposure settings assigned to the imaging system's imaging devices. A weighting factor assigned to each imaging device is influenced by the current state of the vehicle's transmission system. In another implementation, the adaptive display service is configurable to gradually increase the brightness of the graphical representation of subsequent image data over a temporary period after the imaging system has automatically configured for a lower frame rate.In another implementation, the adaptive display service is configurable to progressively decrease the brightness of the graphical representation of the subsequent image data over a temporary period after automatically configuring the imaging system for a higher frame rate. In some implementations, the adaptive display service is configurable to automatically switch the imaging system from a higher to a lower frame rate when the current state of a vehicle transmission system is reverse gear and a redness value associated with the image data is greater than a threshold, provided the estimated ambient light level around the vehicle is below a second threshold. In some implementations, the adaptive display service is configurable to dynamically adjust the brightness of the display device, at least partially, based on the estimated ambient light level.
[0013] A vehicle system is also provided, comprising an illuminance meter for providing measured luminance, a steering system, a display device, an imaging system with a variety of imaging devices, and a control module connected to the illuminance meter, steering system, display device, and imaging system. The control module is configurable to configure the imaging system for an initial frame rate based on the measured luminance, to determine a representative exposure setting for the imaging system at least partially based on a current steering angle associated with the steering system, and to determine an estimated illuminance of an environment at least partially based on the representative exposure setting and image data from the imaging system acquired using the initial frame rate.Automatically configuring the imaging system for a second frame rate, different from the initial frame rate, based on the estimated illuminance; retrieving subsequent image data from the imaging system acquired using the second frame rate; providing a graphical user interface that includes a graphical representation of the subsequent image data on the display device; and setting a display characteristic of the graphical user interface for a temporary period after the automatic configuration of the imaging system for the second frame rate. In one or more embodiments, the vehicle system comprises a transmission system, wherein the control module is configurable to determine the representative exposure setting in a manner influenced by a current state of the transmission system. BRIEF DESCRIPTION OF THE FIGURES
[0014] The exemplary aspects are described below in conjunction with the following figures, where identical numbers denote identical elements and where: Fig. Figure 1 is a block diagram showing a vehicle in accordance with different embodiments; Fig. Figure 2 is a flowchart illustrating an adaptive camera display process for use with the vehicle's system. Fig. 1 is suitable in accordance with various implementations; and Fig. 3 is a flowchart illustrating a process for selecting the frame rate, which is used with the vehicle system of Fig. 1 in conjunction with the adaptive camera display process of Fig. 2 is suitable according to one or more of the implementations described herein. DETAILED DESCRIPTION
[0015] The following detailed description is merely exemplary and is not intended to limit its application and use. Furthermore, it is not intended to be bound by any express or implied theory set forth in the preceding introduction, summary, or the following detailed description. As used herein, the term "module" refers to any hardware, software, firmware, electronic control component, processing logic, and / or processing device, individually or in any combination, including, but not limited to: application-specific integrated circuits (ASICs), an electronic circuit, a processor (common, dedicated, or as a group), and memory executing one or more software or firmware programs, a combinational logic circuit, and / or other suitable components providing the described functionality.
[0016] Fig. Figure 1 shows an exemplary implementation of a vehicle 100 comprising an imaging system 102 that provides image data to display software in a control module 104, which renders, generates, or otherwise provides a graphical user interface (GUI) on a display device 106 containing one or more graphical representations of the image data acquired by the imaging system 102. In one or more implementations, the imaging system 102 is, for example, realized as a surround-view camera system comprising a front camera configured to capture an area in front of the vehicle 100 (e.g., a forward-facing camera mounted in a radiator grille of the vehicle), a rear-view camera configured to capture an area behind the vehicle 100, and side cameras configured to capture areas to the sides of the vehicle 100 (e.g.,(two side cameras mounted on the side mirrors). In such implementations, the display software may be able to provide surround-vision GUIs, including, but not limited to, an overhead, bird's-eye, and / or shell view of the vehicle 100 in the context of its surroundings; a rear-view camera perspective when the vehicle 100 is engaged in reverse; a front-view camera perspective when the vehicle 100 is engaged in driving mode; a front or rear top-down view (e.g., as part of a low-speed parking assistance system); a perspective front or rear view of the vehicle 100; a front or rear side view from outside the vehicle 100; and / or similar. In some embodiments, the image data may be used for surround-vision video recordings that can be stored in the vehicle 100 or uploaded to a remote location (e.g., cloud storage).
[0017] In exemplary implementations, Vehicle 100 is realized as a motor vehicle, such as a passenger car, a sport utility vehicle (SUV), a pickup truck, or similar, but it should be recognized that the subject matter described here is not limited to motor vehicles and can be implemented in an equivalent manner in connection with any other vehicle, including motorcycles, trucks, recreational vehicles (RVs), watercraft, aircraft, and the like. It should be noted that Fig. 1 shows a simplified representation of a vehicle 100 for explanatory purposes and is not intended to be restrictive in any way.
[0018] The imaging system 102 generally represents a combination of one or more cameras, image sensors or other imaging devices configured to capture, perceive or otherwise obtain image data representative of an environment near the vehicle 100 within a field of view assigned to the respective camera or imaging device of the imaging system 102.In exemplary implementations, the imaging system 102 includes, for example, a forward-facing camera mounted on the front of the vehicle 100 to capture image data for a front area of the environment in front of the vehicle 100, a rear-facing camera mounted on the rear of the vehicle 100 to capture image data for an area of the environment behind the vehicle 100, and side cameras mounted on the side mirrors or other locations on the side of the vehicle 100 to capture image data for the environment in laterally adjacent areas.In one or more implementations, the imaging system 102 also includes one or more controllers or processors connected to the imaging devices to support communication with the control module 104 and to facilitate the management or control of one or more properties, parameters, or other settings of the imaging devices by the control module 104. As described in more detail below, the control module 104 can, for example, interact with a controller of the imaging system 102 to control or otherwise adjust the exposure time or other exposure settings, frame rates, camera gain factors, and / or the like.
[0019] The control module 104 generally represents the combination of hardware, software, firmware, processing logic and / or other components connected to the vehicle 100 and configured to synthesize and process the image data from the imaging system 102 to generate GUIs on the display device 106 containing graphical representations of the area of the environment near the vehicle 100, as described in more detail below.In exemplary implementations, the control module 104 is also configurable to support an adaptive camera mode and a display mode, in which an adaptive display service on the control module 104 automatically adjusts one or more settings associated with the imaging devices of the imaging system 102, the display device 106 and / or the GUIs to adjust the frame rate, brightness or other properties of the image data (or graphical representations thereof) to improve the visibility of the GUIs on the display device 106 in a manner that compensates for the current ambient lighting, as measured by an illuminance sensor element 108 (e.g., an ambient light sensor or lux meter), and the current state of the vehicle 100, as displayed by one or more on-board systems 110.
[0020] In exemplary implementations, the control module 104 comprises at least one processor and one non-transferable, computer-readable storage device or medium. The processor can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors connected to the control module 104, a semiconductor-based microprocessor (in the form of a microchip or chipset), a microprocessor, any combination thereof, or, more generally, any instruction-executing device. The computer-readable storage devices or media can include volatile and non-volatile memory, such as read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM).KAM is a persistent or non-volatile memory that can be used to store various operating variables while the processor is powered off. The computer-readable memory device(s) can be implemented using any number of known memory devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combined memory devices capable of storing data, some of which are executable instructions used by Control Module 104. The instructions may comprise one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions.When the instructions are executed by the processor, they cause the control module 104 to receive and process signals and / or data from the imaging system 102, the illuminance measuring element 108 and / or the on-board systems 110 and to provide an adaptive display service that can be configured to perform logic, calculations, procedures and / or algorithms for setting one or more properties, parameters or other settings associated with the imaging system 102 and / or the display device 106.
[0021] In exemplary implementations, the display device 106 is realized as an electronic display device located on board the vehicle 100 or otherwise connected to another system on board the vehicle 100, such as any type of infotainment module, navigation head unit, or other similar or suitable unit located on board the vehicle 100, which may be integrated into a dashboard or other console within a passenger compartment of the vehicle 100. In such implementations, the display device 106 may be coupled to the control module 104 via a bus interface or similar communication interface, which includes the hardware, software, firmware, processing logic, and / or other components for communication with / from the display device 106, such as...a processor, microcontroller or other electronics that supports communication via a Controller Area Network (CAN) bus or other wired vehicle network.
[0022] In other embodiments, the display device 106 can be implemented as an electronic device assigned to a vehicle owner or another user associated with the vehicle 100, which is separate and distinct from the vehicle 100 but communicatively coupled with the control module 104, such as a smartphone, a desktop computer, a mobile computer (e.g., a tablet computer, a laptop computer, or a netbook computer), a video game device, a digital media player, a digital camera or video camera, a portable computing device (e.g., a tablet computer, a laptop computer, or a netbook computer), a video game device, a digital media player, a home entertainment device, a digital camera or video camera, a portable computing device (e.g., a smartwatch, smart glasses, smart clothing), or the like.In such implementations, the control module 104 can be coupled to the display device 106 via a telematics system or other communication system configured to wirelessly transmit information to and from the control module 104 via a communication network, such as, but not limited to, a network of other vehicles (“V2V” communication), an infrastructure network (“V2I” communication), a wireless local area network (WLAN), a personal area network (PAN), a cellular network, a dedicated short-range communication channel (DSRC), and / or the like. Although in . Fig. 1. Where the display device 106 is shown as being connected to the vehicle 100 or otherwise arranged on board the vehicle 100, in practice the display device 106 may be separate and distinct from the vehicle 100.
[0023] The on-board systems 110 generally represent the combination of hardware, software, firmware, processing logic, and / or other components connected to the vehicle 100 that are capable of providing the control module 104 with information about the current vehicle status in order to adaptively adjust the imaging system 102 and / or the display device 106, as described in more detail below. For example, the on-board systems 110 may include a navigation system or other positioning system configured to identify or otherwise determine the current heading of the vehicle 100, the current geographic location of the vehicle 100, and / or similar information.Furthermore, the on-board systems 110 may include a steering system capable of providing data and / or information that identifies a current steering wheel angle connected to the steering system of the vehicle 100, and a transmission system capable of providing data and / or information that identifies a current gear or transmission state for the vehicle 100 (e.g. driving, parking, reversing, etc.).
[0024] Fig. Figure 2 shows an exemplary implementation of an adaptive camera display process 200, suitable for implementation by a vehicle-connected control module to automatically adjust properties, parameters, or other settings associated with imaging devices and / or display devices essentially in real time to respond dynamically and adaptively to changes in ambient brightness and / or changes in vehicle state. For illustration, the following description may refer to elements mentioned above in connection with Fig. 1 mentioned. Although parts of the adaptive camera display process 200 can be performed by different elements of a vehicle system, for explanatory purposes the subject can be described here mainly in connection with the adaptive camera display process 200, which is mainly performed by the control module 104.
[0025] With reference to Fig. 2 and with continued reference to Fig. 1. The adaptive camera display process 200 is initialized by retrieving the current ambient luminance from the illuminance sensor 108 and setting the initial frame rate for the imaging system 102 based on the ambient luminance at 202. In this context, the control module 104 receives the current luminance measurement, either output by the lux meter or another illuminance sensor 108 on board the vehicle 100, or obtains it by other means, and then sets the initial frame rate for the imaging system 102 based on the current luminance. For example, the illuminance sensor 108 can be implemented as a lux meter positioned on or around the dashboard or windshield of the vehicle 100 to measure the ambient illuminance at a point in front of the driver of the vehicle 100.When the current ambient illuminance is greater than or equal to a threshold, the control module 104 commands, signals, or instructs the imaging system 102 to configure the cameras or other imaging devices for a high frame rate operating mode (e.g., 30 frames per second). Conversely, when the current ambient illuminance is below the threshold, the control module 104 commands, signals, or instructs the imaging system 102 to configure the cameras or other imaging devices for a low frame rate operating mode (e.g., 15 frames per second) to allow for longer exposure times.
[0026] After initializing the frame rate operating mode for the imaging system 102, the adaptive camera display process 200 analyzes the image data acquired by the imaging system 102 to determine an estimated ambient illuminance for the surroundings based on the image data and the current vehicle dynamics, and dynamically adjusts the frame rate operating mode of the imaging system 102 in response to changes in the estimated ambient illuminance. In exemplary implementations, the adaptive camera display process 200 receives current vehicle state information at 204 or obtains it by other means and then calculates or determines a representative exposure setting for the fused surround-view image or other fusion of image data from the individual cameras or imaging devices based on the current vehicle state at 206.In this respect, the adaptive camera display process 200 uses the current state of the vehicle transmission and the current steering wheel angle, which is linked to the vehicle steering system, to determine which camera(s) are most relevant for the current operation of the vehicle 100, and then calculates the representative exposure setting as a weighted average of the exposure settings of the individual cameras, with the weighting factors being dynamically determined based on the steering wheel angle and the transmission state to preferentially weight the camera(s) most relevant for the current vehicle operation. In this context, the representative exposure setting (e.g. D ) can be represented by the following equation e D = Σ i w i e i are represented, where e i for the exposure setting of a respective camera (i) and w irepresents the weighting factor assigned to the respective camera, where ∑ i w i = 1.
[0027] In an implementation where the imaging system 102 comprises four cameras (e.g., one forward, one rear, and two side cameras), when the transmission of the vehicle 100 is in motion, meaning that the forward operation of the vehicle 100 is most important, the weighting factors assigned to the respective cameras can be determined by the following set of equations: w^front=max(0.4,cos(SWA)); w^right={min(0.2,cos(90−SWA)),if SWA>00.1,if SWA≤0; w^left={min(0.2,cos(−SWA)),if SWA≤00.1,if SWA>0 and w^rear=0.1, where SWA represents the current steering wheel angle and the values of the weighting factors are normalized so that the sum of the respective weighting factors equals 1, according to the equation wi=wl^∑iwl^. In this respect, the adaptive camera display process 200 preferentially weights the exposure setting of the front camera when the transmission is in motion, and increasingly weights one of the left and right side cameras in accordance with the steering wheel angle, while neglecting the exposure setting of the rear camera and the other side camera relative to the steering wheel angle based on the angle's magnitude. Similarly, the weighting factors assigned to the respective cameras can be determined by the following set of equations when the vehicle's transmission 100 is in reverse, indicating that the reverse operation of the vehicle 100 is of paramount importance: w^rear=max(0.4,cos(SWA)); w^left={min(0.2,cos(90−SWA)),if SWA>00.1,if SWA≤0; w^right={min(0.2,cos(−SWA)),if SWA≤00.1,if SWA>0 and w^front=0.1.
[0028] After determining the representative exposure setting, the adaptive camera display process 200 identifies or determines one or more areas of interest within the image data acquired by the imaging system 102 at 208 and then calculates or otherwise determines an estimated ambient illuminance based on the image data corresponding to the identified areas of interest, using the representative exposure setting at 210. In exemplary implementations, the control module 104 filters the image data from the various cameras of the imaging system 102 to exclude or otherwise remove subsets or regions of the image data corresponding to the sky, shadows, or other visual artifacts or distortions (e.g., dark regions around the periphery of the camera's field of view due to the focal length of the camera lens, such as...).(e.g., a fisheye lens), which are not representative of the illuminance of the vehicle's surroundings. After filtering out subsets of image data corresponding to the non-surrounding areas, exemplary implementations convert the remaining subsets of image data corresponding to the surrounding areas of interest from RGB color model component values to corresponding YUV color model component values before averaging the Y-channel luminance component values over the identified surrounding areas of interest is calculated or otherwise determined. In this respect, the average Y-channel luminance component provides a robust average brightness metric of the vehicle's surroundings for applications such as the camera system's auto-exposure / auto-gain algorithm, the adaptive display of correct brightness algorithm, the headlight / taillight brightness adjustment algorithm, etc.
[0029] In exemplary implementations, the adaptive camera display process 200 uses the estimated ambient illuminance based on the image data to dynamically determine whether the frame rate setting for the imaging system 102 should be adjusted based on changes in the environment and different vehicle dynamics at 212, and commands, signals, instructs or otherwise configures the imaging system 102 for the different frame rate at 214 when the estimated illuminance indicates that the frame rate should be changed.In this context, if the current frame rate setting of the imaging system 102 corresponds to the low frame rate operating mode, the control module 104 can automatically determine to set the frame rate setting of the imaging system 102 to the high frame rate operating mode if the estimated ambient illuminance of the image is greater than a threshold value, and give the imaging system 102 corresponding commands or instructions to configure the cameras of the imaging system 102 for the high frame rate operating mode.Similarly, if the current frame rate setting of the imaging system 102 corresponds to the high frame rate operating mode, the control module 104 can automatically determine to set the frame rate setting of the imaging system 102 to the low frame rate operating mode if the estimated ambient illuminance of the images falls below a threshold, and provide corresponding commands or instructions to the imaging system 102 to configure its cameras for the low frame rate operating mode. In this context, the thresholds for dynamic and adaptive frame rate switching can include an offset to create hysteresis that prevents the imaging system 102 from switching back and forth between different frame rates sequentially.
[0030] Still referring to Fig. 2. In exemplary implementations, when the adaptive camera display process 200 changes the frame rate at the imaging system 102, the adaptive camera display process 200 temporarily smooths the brightness of the subsequent image data acquired at 216 using the different frame rate across successive images over a transition period. In this respect, the change in the camera frame rate can lead to an abrupt change in image brightness, which may affect the ability of the driver or other user to accurately perceive the environment displayed on the display device 106. Accordingly, in exemplary implementations, the control module 104 gradually increases or decreases the image brightness in response to the change in the frame rate setting of the imaging system 102 over the duration of a transition period before smoothing or adjusting the image brightness.In one or more embodiments, the control module 104 uses a multiplier that increases or decreases linearly or semi-linearly over the transition period to linearly smooth the perceived image brightness during a frame rate change.
[0031] For example, in an implementation, when transitioning from a high frame rate operating mode to a low frame rate operating mode, the smoothing multiplier (δ) can be t ), which is used to adjust or scale the image brightness, is determined by the following equation δ t = δ0 + (t - 1) δ step are determined, where t represents the respective frame number for the respective frame of the low frame rate operating mode, until a maximum number of transition frames (t w ) to terminate the smoothing multiplier, δ step represents the amount by which the value of the smoothing multiplier is incrementally adjusted, as defined by the equation δstep=1−δ0tw and δ0 is a constant based on a relationship or ratio between the luminance of the preceding image data, acquired in high frame rate mode, and the luminance of the output image data when transitioning to low frame rate mode. Conversely, when transitioning from low frame rate mode to high frame rate mode, the smoothing multiplier (δ) t ), which is used to adjust or scale the image brightness, can be determined by the following equation δ t = δ0 - (t - 1) δ step are determined, where t represents the respective frame number for the respective frame of the low frame rate operating mode, until a maximum number of transition frames (t w ) to terminate the smoothing multiplier, δ step represents the amount by which the value of the smoothing multiplier is incrementally adjusted, as defined by the equation δstep=1−δ0tw δ0 equals 1.
[0032] Referring again to Fig. 2. In exemplary implementations, the adaptive camera display process 200 also dynamically adjusts and sets the display brightness essentially in real time, based on the estimated illuminance of the current environment using the current vehicle state information at 218. In this context, the control module 104 calculates or determines a brightness setting value based on a weighted average of the estimated illuminance derived from the areas of interest in the image data (determined at 210) and the current illuminance measurement output by the lux meter or other illuminance measuring element 108, with the weighting factors varying according to the current transmission state of the vehicle 100. For example, in one implementation, the adaptive display brightness setting value for the current frame (bt^) as a weighted average of a raw brightness setting value calculated for the current image (b t ) and the raw brightness setting value calculated for the previous image (b t-1 ), which can be determined by the equation b^t=abt+(1−α)bt−1, where α is a weighting factor with a value between zero and one. The raw brightness setting can be calculated based on the merged illuminance estimate for the current image (µ̂) in accordance with equation b. t = bmin+(bmax−bmin)(μ^μcmax)1g to be calculated, where b min represents the minimum raw brightness setting value calculated for a previous image, b max represents the maximum raw brightness setting value calculated for a previous single image, µ cmaxrepresents the maximum estimated illuminance derived from a full frame of image data, g is a calibratable parameter that controls the curvature of the change in the adaptive display brightness setting, and the merged illuminance estimate is calculated based on the estimated illuminance derived from the current full frame of image data and the current luminance measurement.
[0033] In one or more implementations, the fused illuminance estimate for the current image (µ̂) is calculated as a weighted average of the estimated illuminance derived from the current image data frame and the current luminance measurement in a manner that varies depending on the current transfer state according to the equation μ^={0.5μc+0.5l^ if transmission=drive or l>4000.65μc+0.35l^ if transmission≠drive or l≤400 where µc The estimated illuminance is derived from the current full frame of the image data, and l is the current luminance measurement, and l̂ is calculated based on a ratio between the current luminance measurement and a luminance threshold (e.g., 65,000 lumens). In this respect, the merged illuminance estimation can give greater weight to the estimated illuminance derived from the current frame of the image data in dimly lit environments or when the vehicle is reversing, parked, or otherwise not moving in a forward direction corresponding to the front position of the illuminance measuring element 108.
[0034] In one or more exemplary implementations, for the purposes of adaptive adjustment of the display brightness, the estimated illuminance derived from the current image data frame is calculated as a weighted sum of the estimated illuminance derived from the respective image data from each of the cameras of the imaging system 102 according to the equation µ c = ∑ i w i µ i where µ i represents the estimated illuminance assigned to the respective camera, and w i This represents the weighting factor assigned to the respective camera. In example implementations, the weighting factors satisfy the equation ∑ i w i = 1 and w i depending on the relationship between the orientation or field of view of the camera and the current transmission state of the vehicle 100 according to the equation w i = γ i β iwhere γ i represents a transfer-based weighting factor and β iThis represents a weighting factor that corresponds to the viewing angle of each camera in relation to the road. For example, if the current transmission state corresponds to driving mode or another forward gear, the adaptive camera display process 200 can adjust the transmission-based weighting factor to give more weight to the forward camera and less weight to the rear camera. Conversely, if the current transmission state corresponds to reverse gear, the adaptive camera display process 200 can adjust the transmission-based weighting factor to give more weight to the rear camera and less weight to the front camera.The adaptive camera display process 200 can also use the course, navigation map data and possibly other information available from other vehicle systems 110 to calculate, estimate or otherwise determine the road view angle for a particular camera in order to adjust the weighting factor for the view angle increasing or decreasing β. i based on the course of vehicle 100 and the orientation of the respective camera in relation to the course or orientation of the current lane, to adjust increasingly or decreasingly.
[0035] As in the Fig. As shown in Figures 1-2, after determining the adaptive display brightness setting value of the display device 106, the control module 104 automatically issues commands, signals, or other instructions to increase or decrease the brightness of the display device 106 accordingly, in order to adapt to the current illuminance of the vehicle 100 and the surroundings in a manner that takes into account the current transmission state and the direction of travel of the vehicle 100. By using a moving average to determine the adaptive brightness setting value of the display over successive frames, the brightness of the display transitions smoothly, thereby improving the perception of the GUI displays and the associated images shown on the display device 106.
[0036] The loop defined by 204, 206, 208, 210, 212, 214, 216, and 218 can repeat indefinitely during the operation of the vehicle 100 to dynamically adjust the frame rate, image brightness, and display brightness in essentially real time in response to changes in ambient light intensity and / or changes in vehicle condition, thereby enhancing the driver's perception of the images displayed on the display device 106. By smoothly or gradually adjusting the brightness associated with the image data and / or the brightness setting of the display device 106, human factors are improved by avoiding abrupt changes in brightness or perceived luminance that could affect the driver's ability to perceive or assess the environment displayed on the display device 106.
[0037] Fig. Figure 3 shows an exemplary implementation of a frame rate selection process 300, which is suitable for implementation in conjunction with the adaptive camera display process 200 (e.g., in Figures 212 and 214) to automatically change the frame rate based on the estimated or perceived ambient luminance in essentially real time, in order to respond dynamically and adaptively to changes in ambient brightness and / or changes in vehicle state. For illustrative purposes, the following description may refer to elements mentioned above in connection with Fig. 1 mentioned. Although parts of the frame rate selection process 300 can be performed by different elements of a vehicle system, for explanatory purposes the subject is mainly described here in connection with the frame rate selection process 300, which is mainly performed by the control module 104.
[0038] The frame rate selection process 300 is initialized by determining, based on the current luminance measured by the illuminance sensor 108 at 302, whether the imaging system 102 should be configured for a higher frame rate or a lower frame rate operating mode. If the current measured luminance value is greater than or equal to an initial selection threshold, the frame rate selection process 300 automatically configures the imaging system 102 for the higher frame rate operating mode at 304. Conversely, if the current measured luminance value is less than the initial selection threshold (e.g., l), the process does not configure the imaging system 102 for the higher frame rate operating mode. <l0), konfiguriert der Bildratenauswahlprozess 300 das Bildgebungssystem 102 automatisch für den Betriebsmodus mit niedrigerer Bildrate bei 314.
[0039] After configuring the imaging system 102 for higher frame rate operating mode, the frame rate selection process 300 calculates or determines an estimated ambient illuminance based on the image data (e.g., at 210) and then checks whether the estimated illuminance, based on the acquired image data, is above a lower luminance threshold for switching frame rates at 306 for the areas of interest. It then automatically switches from higher frame rate operating mode to lower frame rate operating mode if the estimated illuminance is below the lower luminance threshold (e.g., l). <l1). In diesem Zusammenhang kann der untere Leuchtdichteschwellenwert für das Umschalten von der Betriebsart mit höherer Bildrate auf die Betriebsart mit niedrigerer Bildrate geringer sein als der anfängliche Auswahlschwellenwert (z. B.l1 <l0), um unerwünschte Übergänge oder ein Umschalten zwischen den Betriebsarten mit höherer Bildrate zu verhindern. Auf diese Weise schaltet das Steuermodul 104 das Bildgebungssystem 102 automatisch von der Betriebsart mit höherer Bildrate in die Betriebsart mit niedrigerer Bildrate um, wenn die aus den erfassten Bilddaten abgeleitete geschätzte Beleuchtungsstärke einen ausreichenden Rückgang der wahrgenommenen Umgebungsbeleuchtung der Umgebung anzeigt.
[0040] The frame rate selection process 300 also checks whether the estimated illuminance is above an intermediate luminance threshold at 308, and if the estimated illuminance is below the intermediate luminance threshold (e.g., l <l2), bestimmt der Bildratenauswahlprozess 300 bei 310, ob die wahrgenommene Beleuchtungsstärke auf die Fahrzeugbremslichter zurückzuführen ist. In diesem Zusammenhang kann der mittlere Leuchtdichteschwellenwert größer sein als der untere Schwellenwert für das automatische Umschalten von der Betriebsart mit höherer Bildrate auf die Betriebsart mit niedrigerer Bildrate (z.B. l1<l2), wobei der Bildratenauswahlprozess 300 den aktuellen Übertragungszustand des Fahrzeugs 100 und den Grad der Rötung in den erfassten Bilddaten analysiert, um zu überprüfen, ob die Differenz zwischen der geschätzten Beleuchtungsstärke und dem unteren Leuchtdichteschwellenwert nicht nur auf die Fahrzeugbremslichter zurückzuführen ist.In one implementation, for example, the frame rate selection process 300 calculates or determines a redness value for the acquired image data (e.g., a percentage of the red component relative to other RGB components) and then determines that the estimated illuminance is attributable to the vehicle's brake lights when the current transmission status is reverse and the redness value is greater than a threshold percentage (e.g., 30%). In this context, the control module 104 automatically switches the imaging system 102 from higher frame rate mode to lower frame rate mode when the estimated illuminance derived from the acquired image data is relatively low and more likely attributable to the vehicle's brake lights than to ambient lighting.
[0041] If the estimated illuminance is lower than the mean luminance threshold and is not due to the vehicle's brake lights, the frame rate selection process 300, in exemplary implementations, also checks or confirms that the brightness of each respective image captured by a respective camera or imaging device of the imaging system 102 is greater than a brightness threshold at 312. If the brightness of any of the images from the forward, rear, or side-view camera is below the brightness threshold, the frame rate selection process 300 can automatically switch the imaging system 102 from the higher frame rate operating mode to the lower frame rate operating mode to increase the likelihood that each of the images captured by the respective cameras of the imaging system 102 has adequate brightness for display on the display device 106.
[0042] If the estimated illuminance of the captured image data is maintained above the lower and middle luminance thresholds, the frame rate selection process 300 maintains the higher frame rate operating mode (e.g., at 212) and allows the adaptive camera display process 200 to dynamically adjust the display brightness based on the estimated illuminance of the current environment as perceived in the higher frame rate operating mode at 218. Conversely, if the estimated illuminance of the captured image data falls below the lower luminance threshold at 306, if one of the captured images falls below a brightness threshold at 312, or if the estimated illuminance of the captured image data is attributable to the vehicle's brake lights, the frame rate selection process 300 automatically switches the imaging system 102 from the higher frame rate operating mode to the lower frame rate operating mode at 314 (e.g.,at 214 of the adaptive camera display process 200), (e.g., at 214 of the adaptive camera display process 200), which initiates the transition period for the adaptive camera display process 200, smoothing the perceived brightness of the image data subsequently captured in the lower frame rate operating mode at 216. Thus, when the frame rate selection process 300 automatically changes the frame rate operating mode from the higher frame rate mode to the lower frame rate mode, the adaptive camera display process 200 smooths the perceived brightness to avoid an abrupt change in brightness that might otherwise occur due to the longer exposure time associated with the lower frame rate.
[0043] In a similar manner to that described in connection with the higher frame rate operating mode, the frame rate selection process 300, after the configuration of the imaging system 102 for the lower frame rate operating mode, calculates or determines an estimated ambient illuminance based on the image data (e.g., at 210) and then checks whether the estimated illuminance based on the acquired image data for the regions of interest does not exceed an upper luminance threshold for switching frame rates at 316 (see Fig.3) In this context, the upper luminance threshold is greater than the mid-range luminance threshold by a certain offset (e.g., Δl) to create hysteresis that prevents switching between different frame rates. If the estimated illuminance is greater than an upper luminance threshold (e.g., l > l² + Δl), the frame rate selection process 300 automatically switches the imaging system 102 from the lower frame rate mode to the higher frame rate mode.
[0044] In exemplary implementations, the frame rate selection process 300 also checks or confirms whether the brightness of each respective image captured by a respective camera or imaging device of the imaging system 102 is greater than a brightness threshold at 318. If the brightness of each of the images from the front, rear, or side-view camera is above the brightness threshold, the frame rate selection process 300 can automatically switch the imaging system 102 from the lower frame rate operating mode to the higher frame rate operating mode to improve the visibility of the captured images.
[0045] As described above, when the frame rate selection process 300 automatically switches the imaging system 102 from the lower frame rate mode to the higher frame rate mode at 214, the transition period for the adaptive camera display process 200 is initialized. This process smooths the perceived brightness of the image data subsequently acquired in the higher frame rate mode at 216. Therefore, when the frame rate selection process 300 automatically switches from the lower frame rate mode to the higher frame rate mode, the adaptive camera display process 200 smooths the perceived brightness to avoid an abrupt stop-down that might otherwise occur due to the shorter exposure time associated with the higher frame rate.
[0046] Through automatic and adaptive transitions between different frame rate operating modes and the corresponding smoothing of the resulting image data, the vehicle 100 can switch from bright to dark environments or vice versa without impairing the driver's perception of the graphical user interfaces displayed on the display device 106. Furthermore, the luminance or brightness of the captured image data and the corresponding GUIs displayed on the display device 106 can be adjusted to the perceived environment in a manner that reflects the current operation of the vehicle 100 (e.g., the current state of the transmission, the current steering wheel angle, the current heading of the vehicle, and / or similar factors) as well as the ambient brightness of the area of interest.For example, surround-view camera systems and fused images can play an important role in parking assistance and other vision and perception applications, but are susceptible to fluctuations in lighting due to physical phenomena (e.g., twilight) or when a vehicle enters or exits a different environment (e.g., a garage). The device described here allows surround-view camera systems to quickly adapt to these types of fluctuations while avoiding abrupt changes in display brightness that could otherwise impair perception and road safety.Using the steering wheel angle, transmission status, and potentially other factors, the device described here is able to reliably determine the ambient brightness around a host vehicle through localized analysis, focusing on the areas most likely to be of interest given the current vehicle state. This improves the driver's view of the surroundings and enhances road safety by rapidly adjusting the display's image and brightness to changes in lighting conditions. This improves perception and eye comfort, which in turn boosts driver confidence and situational awareness, allowing for more coherent vision across multiple views and image frames.
[0047] For the sake of brevity, conventional techniques related to lane keeping assist, driver assistance functions, autonomous vehicles, and other functional aspects of the systems (and the individual operating components of the systems) are not described in detail here. Furthermore, the connecting lines shown in the various figures are intended to represent exemplary functional relationships and / or physical connections between the different elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in an implementation of the subject.
[0048] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Therefore, any implementation described herein as "exemplary" is not necessarily to be understood as preferred or advantageous over other implementations. All embodiments described herein are exemplary embodiments intended to enable the person skilled in the art to manufacture or use the invention and are not intended to limit the scope of the invention as defined by the claims.
[0049] Experts know that the various logical blocks, modules, circuits, and algorithm steps described in connection with the implementations presented here can be implemented as electronic hardware, computer software, or combinations of both. Some of the implementations are described above in the form of functional and / or logical block components (or modules) and various processing steps. However, it should be acknowledged that such block components (or modules) can be realized by any number of hardware, software, and / or firmware components configured to perform the specified functions. To illustrate this interchangeability of hardware and software, various components, blocks, modules, circuits, and steps have been described above in general terms with regard to their functionality.Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Experts may implement the described functionality in different ways for each individual application, but such implementation decisions should not be interpreted as a departure from the scope of the present invention.
[0050] When implemented in software or firmware, the various elements of the systems described here are essentially the code segments or instructions that perform the different tasks. The program or code segments can be stored on a processor-readable medium or transmitted via a transmission medium or communication path using a computer data signal embodied in a carrier wave. "Computer-readable medium," "processor-readable medium," or "machine-readable medium" can encompass any medium capable of storing or transmitting information. Examples of processor-readable medium include an electronic circuit, semiconductor memory, ROM, flash memory, erasable ROM (EROM), a floppy disk, a CD-ROM, an optical disc, a hard disk, a fiber optic medium, a radio frequency connection, or similar devices.A computer data signal can encompass any signal capable of propagating through a transmission medium, such as electronic network channels, optical fibers, air, electromagnetic paths, or RF connections. Code segments can be downloaded via computer networks such as the internet, an intranet, a LAN, or similar.
[0051] In this document, relational terms such as "first" and "second," and the like, may be used merely to distinguish one unit or action from another, without necessarily requiring or implying any actual relationship or sequence between such units or actions. Numerical ordinal numbers such as "first," "second," "third," etc., merely denote different individuals from a multitude and do not imply any sequence unless expressly defined in the claims. The order of the text in any of the claims does not mean that the process steps must be performed in a chronological or logical order corresponding to that order, unless expressly determined by the language of the claim.The process steps can be exchanged in any order without deviating from the scope of the invention, provided that such an exchange does not contradict the claim text and is logically coherent. Furthermore, the foregoing description may refer to elements, nodes, or features that are "coupled" to one another. Unless expressly stated otherwise, "coupled" here means that one element / node / feature is directly or indirectly connected to (or communicates with) another element / node / feature, and not necessarily mechanically. For example, two elements may be coupled to one another physically, electronically, logically, or in any other way via one or more additional elements.Although the drawings show an exemplary arrangement of elements that are directly connected, additional elements, devices, features, or components may be present in an implementation of the depicted object. Furthermore, certain terms are used here for reference purposes only and should therefore not be considered restrictive.
[0052] Although at least one exemplary aspect has been presented in the preceding detailed description, it should be understood that there are a large number of variations. It should also be noted that the exemplary aspect or aspects are merely examples and are not intended to limit the scope, applicability, or configuration of the disclosure in any way. Rather, the preceding detailed description is intended to provide the person skilled in the art with practical guidance for implementing the exemplary aspect or aspects. It is understood that various modifications in the function and arrangement of the elements can be made without departing from the scope of the disclosure as set forth in the attached claims and their statutory equivalents.
Claims
[1] A method (300) to support the operation of a vehicle, the method comprising: image data is received, by means of a control module (104) connected to the vehicle, from an imaging system (102) on board the vehicle, wherein the image data is acquired at the imaging system (102) using an initial frame rate; Determine, by the control module (104), current vehicle status information; Determine, by the control module, a representative exposure setting for the imaging system (102) that is based at least partially on the current vehicle status information; Determine, by the control module (104), an estimated illuminance of the vehicle's surroundings, based at least partially on the image data and the representative exposure setting; automatic configuration, by the control module (104), of the imaging system (102) for a second frame rate that differs from the initial frame rate, based on the estimated illuminance; and thereafter: Received from subsequent image data from the imaging system (102) acquired at the second frame rate; Providing a graphical user interface with a graphical representation of the subsequent image data on a display device connected to the vehicle (106); and Setting a display characteristic of the graphical user interface over a temporary period of time after the automatic configuration of the imaging system (102) for the second frame rate, wherein setting the display characteristic includes stepwise adjusting the brightness of the graphical representation of the subsequent image data over the temporary period of time, wherein the stepwise adjustment of brightness comprises configuring the brightness of the graphical representation of an initial image of the subsequent image data based on a weighted average of the initial image of the subsequent image data captured at the second frame rate and a previous image of the image data captured at the initial frame rate, which further includes the identification of one or more areas of interest within the image data by the control module (104), including the determination of the estimated illuminance: Converting RGB component values for one or more areas of interest into YUV component values; and Averaging the Y-channel luminance component values of the YUV component values over the one or more areas of interest, wherein determining the representative exposure setting comprises calculating the representative exposure setting as a weighted sum of the respective exposure settings associated with the respective imaging devices of the imaging system (102), wherein a respective weighting factor associated with a respective imaging device is influenced by a current state of a transmission system connected to the vehicle, Check (318) whether the brightness of each respective image captured by a respective camera or imaging device of the imaging system (102) is greater than a brightness threshold value, if the brightness of each of the images from a forward, backward or side view camera is above the brightness threshold, automatic switching of the imaging system (102) from a lower frame rate operating mode to a higher frame rate operating mode. [2] Method according to claim 1, wherein the current vehicle state information comprises a current state of a transmission system of the vehicle, wherein determining the representative exposure setting comprises calculating the representative exposure setting as a weighted sum of respective exposure settings assigned to the respective imaging devices of the imaging system, wherein a respective weighting factor assigned to a respective imaging device is influenced by the current state of the transmission system assigned to the vehicle. [3] Method according to claim 1, wherein adjusting the display characteristic comprises gradually increasing the brightness of the graphic representation of the subsequent image data over the temporary period after automatically configuring the imaging system for a lower frame rate. [4] Method according to claim 1, wherein adjusting the display characteristic comprises gradually reducing the brightness of the graphic representation of the subsequent image data over the temporary period after the automatic configuration of the imaging system for a higher frame rate.
Citation Information
Patent Citations
Imaging device, imaging / display method and imaging / display program
EP3410702A1
Vehicle vision system with enhanced low light capabilities
US20140354811A1
Near-eye display system and display method thereof
US20200096769A1