Vehicle system and method for environment camera view display adaptation

Abstract

The invention provides a vehicle, a related system and an adaptive display adjustment method. A method involves obtaining image data captured using an initial frame rate from an imaging system onboard a vehicle; obtaining current vehicle state information; determining a representative exposure setting of the imaging system based at least in part on the current vehicle state information; determining an estimated illuminance of the surroundings of the vehicle based at least in part on the image data and the representative exposure setting; and automatically configuring the imaging system to obtain different updated frame rates based on the estimated illuminance. Subsequently, the method continues by obtaining subsequent image data captured using an updated frame rate; providing, on a display device associated with the vehicle, a graphical user interface including a graphical representation of the subsequent image data; and dynamically adjusting the display characteristics after configuring the imaging system to obtain the updated frame rate.

Description

Technical Field

[0001] The technical field generally relates to vehicle systems and, more particularly, to automatic camera view display adjustments based on current surroundings to improve visibility. Background Art

[0002] Modern vehicles include a variety of enhanced or advanced features to support human drivers in operating the vehicle. Many vehicles are able to sense their environment and facilitate vehicle operation. For example, many vehicles are equipped with many different cameras to provide video or images of different areas outside the vehicle, such as a backup camera or rear view camera, a side view camera, a surround view camera, etc. The video or images captured by the on-board camera are typically displayed on a dashboard display (such as an infotainment display or other center console display). The displayed video or images help the driver operate the vehicle, for example, by facilitating parking or other manual operation of the vehicle when the desired external area is visible.

[0003] In practice, camera systems and corresponding infotainment displays are subject to lighting variations due to the position of the sun (e.g., at dusk and / or dawn) or shadows from other external objects that affect scene lighting outside the vehicle and / or inside the vehicle, such as, for example, infrastructure features (e.g., overpasses, garages, etc.), buildings, trees, geographic features, etc. This results in perceived variations in display brightness, which in turn reduces driver visibility of outside areas, thereby reducing the efficacy of the display. Therefore, it is desirable to provide an adaptive camera and display system to improve visibility and safety independent of lighting or environmental variations. Summary of the invention

[0004] An apparatus for a vehicle and a related method and vehicle system are provided. A method of assisting the operation of a vehicle involves: a control module associated with the vehicle obtains image data captured using an initial frame rate from an imaging system on the vehicle; obtains current vehicle status information associated with the vehicle; determines a representative exposure setting for the imaging system based at least in part on the current vehicle status information; determines an estimated illuminance of the vehicle's surroundings based at least in part on the image data and the representative exposure setting; and automatically configures the imaging system to obtain a second frame rate different from the initial frame rate based on the estimated illuminance. Subsequently, the method: obtains subsequent image data captured using the second frame rate from the imaging system; provides a graphical user interface including a graphical representation of the subsequent image data on a display device associated with the vehicle; and adjusts display characteristics of the graphical user interface for a temporary period of time after automatically configuring the imaging system to obtain the second frame rate.

[0005] In some implementations, adjusting the display characteristic involves gradually adjusting the brightness of the graphical representation of subsequent image data over a temporary period of time. In one or more implementations, gradually adjusting the brightness involves configuring the brightness of the graphical representation of an initial frame of subsequent image data based on a weighted average of the initial frame of subsequent image data captured using a second frame rate and a previous frame of image data captured using an initial frame rate. In some implementations, the method identifies one or more regions of interest within the image data, wherein determining the estimated illumination involves: converting RGB component values ​​of the one or more regions of interest into YUV component values; and averaging Y channel luminance component values ​​of the YUV component values ​​over the one or more regions of interest. In one or more implementations, determining a representative exposure setting involves calculating the representative exposure setting as a weighted sum of corresponding exposure settings associated with corresponding imaging devices of the imaging system, wherein the corresponding weighting factors associated with the corresponding imaging devices are affected by a current state of a transmission system associated with the vehicle.

[0006] In one or more implementations, the current vehicle state information includes a current state of a transmission system of the vehicle, wherein determining a representative exposure setting involves calculating the representative exposure setting as a weighted sum of corresponding exposure settings associated with corresponding imaging devices of the imaging system, wherein the corresponding weighting factors associated with the corresponding imaging devices are affected by the current state of the transmission system associated with the vehicle. In one or more implementations, adjusting the display characteristics involves gradually increasing the brightness of the graphical representation of subsequent image data within a temporary period of time after automatically configuring the imaging system to obtain a lower frame rate. In other implementations, adjusting the display characteristics involves gradually decreasing the brightness of the graphical representation of subsequent image data within a temporary period of time after automatically configuring the imaging system to obtain a higher frame rate. In yet other implementations, automatically configuring the imaging system to obtain a second frame rate involves: when an estimated illumination of the surrounding environment of the vehicle is below a second threshold, when the current state of the transmission system of the vehicle corresponds to reverse and the red score associated with the image data is greater than the threshold, automatically switching the imaging system from the higher frame rate to the lower frame rate. In other implementations, adjusting the display characteristics of the graphical user interface involves dynamically adjusting the brightness of the display device based at least in part on the estimated illumination.

[0007] Also provided is an apparatus for a non-transitory computer-readable medium. The non-transitory computer-readable medium has executable instructions stored thereon, which, when executed by a processor, cause the processor to provide an adaptive display service, which can be configured to: obtain image data captured using an initial frame rate from an imaging system on a vehicle; obtain current vehicle state information associated with the vehicle; determine a representative exposure setting for the imaging system based at least in part on the current vehicle state information; determine an estimated illuminance of the vehicle's surroundings based at least in part on the image data and the representative exposure setting; and automatically configure the imaging system to obtain a second frame rate different from the initial frame rate based on the estimated illuminance. Subsequently, the instructions cause the processor to: obtain subsequent image data captured using the second frame rate from the imaging system; provide a graphical user interface including a graphical representation of the subsequent image data on a display device associated with the vehicle; and adjust display characteristics of the graphical user interface for a temporary period of time after automatically configuring the imaging system to obtain the second frame rate.

[0008] In one implementation, the adaptive display service may be configured to gradually adjust the brightness of the graphical representation of the subsequent image data over a temporary period of time. In another implementation, the adaptive display service may be configured to identify one or more regions of interest within the image data, wherein determining the estimated illumination involves: converting RGB component values ​​of the one or more regions of interest into YUV component values; and averaging the Y channel brightness component values ​​of the YUV component values ​​over the one or more regions of interest. In another implementation, the adaptive display service may be configured to obtain a current state of a transmission system of the vehicle, wherein determining a representative exposure setting involves calculating the representative exposure setting as a weighted sum of corresponding exposure settings associated with corresponding imaging devices of the imaging system, wherein the corresponding weighting factors associated with the corresponding imaging devices are affected by the current state of the transmission system associated with the vehicle. In another implementation, the adaptive display service may be configured to gradually increase the brightness of the graphical representation of the subsequent image data over a temporary period of time after automatically configuring the imaging system to obtain a lower frame rate. In another implementation, the adaptive display service may be configured to gradually decrease the brightness of the graphical representation of the subsequent image data over a temporary period of time after automatically configuring the imaging system to obtain a higher frame rate. In some implementations, the adaptive display service may be configured to automatically switch the imaging system from a higher frame rate to a lower frame rate when the estimated illumination of the vehicle's surroundings is below a second threshold, when the current state of the vehicle's transmission system corresponds to reverse and the red score associated with the image data is greater than the threshold. In some implementations, the adaptive display service may be configured to dynamically adjust the brightness of the display device based at least in part on the estimated illumination.

[0009] A vehicle system is also provided, comprising: an illumination sensing element for providing a measured brightness; a steering system; a display device; an imaging system, including a plurality of imaging devices; and a control module, the control module being coupled to the illumination sensing element, the steering system, the display device, and the imaging system. The control module may be configured to: configure the imaging system based on the measured brightness to obtain an initial frame rate; determine a representative exposure setting for the imaging system based at least in part on a current steering angle associated with the steering system; determine an estimated illumination of the surrounding environment based at least in part on the representative exposure setting and image data from the imaging system captured using the initial frame rate; automatically configure the imaging system based on the estimated illumination to obtain a second frame rate different from the initial frame rate; obtain subsequent image data captured using the second frame rate from the imaging system; provide a graphical user interface including a graphical representation of the subsequent image data on the display device; and adjust display characteristics of the graphical user interface for a temporary period of time after automatically configuring the imaging system to obtain the second frame rate. In one or more implementations, the vehicle system includes a transmission system, wherein the control module may be configured to determine the representative exposure setting in a manner affected by a current state of the transmission system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Exemplary aspects will be described below with reference to the following drawings, wherein like reference numerals refer to like elements, and wherein:

[0011] Figure 1 is a block diagram illustrating a vehicle according to various implementations;

[0012] Figure 2 is a diagram showing various implementations applicable to Figure 1 A flow chart of an adaptive camera display process for a vehicle system; and

[0013] Figure 3 is a combination of one or more implementations described herein. Figure 2 The adaptive camera display process is applicable to Figure 1 Flowchart of the frame rate selection process for a vehicle system. DETAILED DESCRIPTION

[0014] The following detailed description is merely exemplary in nature and is not intended to limit application and use. In addition, there is no intention to be bound by any express or implied theory presented in the previous introduction, overview or detailed description below. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, alone or in any combination, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions.

[0015] Figure 1 An exemplary implementation of a vehicle 100 is shown that includes an imaging system 102 that provides image data to display software at a control module 104 that renders, generates, or otherwise provides a graphical user interface (GUI) on a display device 106 that includes one or more graphical representations of image data captured by the imaging system 102. For example, in one or more implementations, the imaging system 102 is implemented as a surround view camera system that includes a forward-looking camera configured to capture an area in front of the vehicle 100 (e.g., a forward-looking camera mounted in a grille of the vehicle), a rear-view camera configured to capture an area behind the vehicle 100, and side-view cameras configured to capture areas laterally adjacent to the vehicle 100 (e.g., two side-view cameras mounted on side mirrors). In such implementations, the display software can provide a surrounding vision GUI including, but not limited to, a top, bird's eye, and / or bowl view of the vehicle 100 in the surrounding environment, a rear camera view when the vehicle 100 is shifted into reverse, a front camera view when the vehicle 100 is shifted into drive, a front or rear top view (e.g., as part of a low-speed parking assist system), a front or rear perspective view of the vehicle 100, a front or rear side view of the exterior of the vehicle 100, etc. In some implementations, the image data may be used for a surround video recording that may be stored at the vehicle 100 or uploaded to a remote location (e.g., cloud storage).

[0016] In an exemplary implementation, vehicle 100 is implemented as a motor vehicle, such as a passenger car, a sport utility vehicle (SUV), a pickup truck, etc., but it should be understood that the subject matter described herein is not limited to motor vehicles and can be implemented in an equivalent manner in the context of any other means of transportation, including motorcycles, trucks, recreational vehicles (RVs), marine vessels, airplanes, etc. It should be understood that Figure 1 A simplified representation of vehicle 100 is depicted for purposes of explanation and is not intended to be limiting in any respect.

[0017] The imaging system 102 generally represents a combination of one or more cameras, image sensors, or other imaging devices configured to capture, sense, or otherwise obtain image data representing the environment near the vehicle 100 within a field of view associated with a corresponding camera or imaging device of the imaging system 102. For example, in an exemplary implementation, the imaging system 102 includes a front-view camera mounted at the front of the vehicle 100 to capture image data of a front area of ​​the surrounding environment in front of the vehicle 100, a rear-view camera mounted at the rear of the vehicle 100 to capture image data of an area of ​​the surrounding environment behind the vehicle 100, and a side-view camera mounted on a side mirror or other location on the side of the vehicle 100 to capture image data of the surrounding environment in a lateral adjacent area. In one or more implementations, the imaging system 102 also includes one or more controllers or processors associated with the imaging device to support communication with the control module 104 and facilitate management or control of one or more characteristics, parameters, or other settings of the imaging device by the control module 104. For example, as described in more detail below, the control module 104 can interact with the controller of the imaging system 102 to control or otherwise adjust the exposure time or other exposure settings, frame rate, camera gain factor, etc.

[0018] The control module 104 generally represents a combination of hardware, software, firmware, processing logic, and / or other components associated with the vehicle 100 that is configured to synthesize and process image data from the imaging system 102 to generate a GUI on the display device 106 that includes a graphical representation of an area of ​​the environment near the vehicle 100, as described in more detail below. In an exemplary implementation, the control module 104 may also be configured to support an adaptive camera mode and a display mode in which an adaptive display service at the control module 104 automatically adjusts one or more settings associated with an imaging device of the imaging system 102, the display device 106, and / or the GUI to adapt the frame rate, brightness, or other characteristics of the image data (or its graphical representation) to improve the visibility of the GUI on the display device 106 by compensating for the current lighting of the surrounding environment indicated by an illuminance sensing element 108 (e.g., an ambient light sensor or a lumen meter) and the current state of the vehicle 100 indicated by one or more onboard vehicle systems 110.

[0019] In an exemplary implementation, the control module 104 includes at least one processor and a non-transitory computer-readable storage device or medium. The processor can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of several processors associated with the control module 104, a semiconductor-based microprocessor (in the form of a microchip or chipset), a microprocessor, any combination thereof, or generally any device for executing instructions. The computer-readable storage device or medium can include, for example, volatile and non-volatile storage in a read-only memory (ROM), a random access memory (RAM), and a keep-alive memory (KAM). KAM is a permanent or non-volatile memory that can be used to store various operating variables when the processor is powered off. The computer-readable storage device or medium can be implemented using any of many known memory devices, such as a PROM (programmable read-only memory), an EPROM (electrical PROM), an EEPROM (electrically erasable PROM), a flash memory, or any other electrical, magnetic, optical, or combined memory device used by the control module 104 that can store data, some of which represent executable instructions. The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by the processor, the instructions cause the control module 104 to receive and process signals and / or data from the imaging system 102, the illuminance sensing element 108, and / or the onboard system 110, and provide adaptive display services that may be configured to perform logic, calculations, methods, and / or algorithms for adjusting one or more characteristics, parameters, or other settings associated with at least one of the imaging system 102 and / or the display device 106.

[0020] In an exemplary implementation, the display device 106 is implemented as an electronic display device that is located on the vehicle 100 or otherwise associated with another system on the vehicle 100, such as, for example, any kind of infotainment module, a navigation head unit, or another similar or suitable unit located on the vehicle 100, and that may be integrated into a dashboard or other console within the passenger compartment of the vehicle 100. In such an implementation, the display device 106 may be coupled to the control module 104 via a bus interface or similar communication interface that includes hardware, software, firmware, processing logic, and / or other components for communicating with the display device 106, such as, for example, a processor, microcontroller, or other electronic device that supports communication via a controller area network (CAN) bus or other wired vehicle network.

[0021] In other implementations, the display device 106 may be implemented as an electronic device associated with the vehicle owner or other user associated with the vehicle 100 that is separate and distinct from the vehicle 100 but is communicatively coupled to the control module 104, such as a smart phone, a desktop computer, a mobile computer (e.g., a tablet computer, a laptop computer, or a netbook computer), a video gaming device, a digital media player, a piece of home entertainment equipment, a digital camera or camcorder, a wearable computing device (e.g., a smart watch, smart glasses, smart clothing), etc. In such implementations, the control module 104 may be coupled to the display device 106 via a telematics system or other communication system that is configured to wirelessly communicate information with the control module 104 over a communication network, such as, but not limited to, networks of other vehicles (“V2V” communications), infrastructure networks (“V2I” communications), wireless local area networks (WLANs), personal area networks (PANs), cellular networks, dedicated short range communications (DSRC) channels, etc. In this regard, although Figure 1 The display device 106 is shown as being associated with or otherwise located on the vehicle 100 , but in reality, the display device 106 may be separate and distinct from the vehicle 100 .

[0022] The onboard system 110 generally represents a combination of hardware, software, firmware, processing logic, and / or other components associated with the vehicle 100 that are capable of providing current vehicle state information to the control module 104 for adaptively adjusting the imaging system 102 and / or the display device 106, as described in more detail below. For example, the onboard system 110 may include a navigation system or other positioning system configured to identify or otherwise determine the current direction of travel of the vehicle 100, the current geographic location of the vehicle 100, etc. In addition, the onboard system 110 may include a steering system capable of providing data and / or information identifying a current steering wheel angle associated with a steering system of the vehicle 100 and a transmission system capable of providing data and / or information identifying a current gear or transmission state (e.g., drive, park, reverse, etc.) of the vehicle 100.

[0023] Figure 2 An exemplary implementation of an adaptive camera display process 200 is depicted, which is suitable for implementation by a control module associated with a vehicle to automatically adjust characteristics, parameters, or other settings associated with an imaging device and / or a display device substantially in real time to dynamically and adaptively respond to changes in ambient illumination and / or vehicle state. For purposes of explanation, the following description may refer to the above description in conjunction with Figure 1Although portions of the adaptive camera display process 200 may be performed by different elements of the vehicle system, for purposes of explanation, the subject matter may be described primarily in the context of the adaptive camera display process 200 being primarily performed by the control module 104 .

[0024] refer to Figure 2 And continue to refer to Figure 1 , the adaptive camera display process 200 begins by obtaining the current luminance of the surrounding environment from the illuminance sensing element 108 at 202 and setting an initial frame rate setting of the imaging system 102 based on the ambient luminance. In this regard, the control module 104 receives or otherwise obtains a current luminance measurement output by a illuminance meter or other illuminance sensing element 108 on the vehicle 100 and then sets the initial frame rate of the imaging system 102 based on the current luminance. For example, the illuminance sensing element 108 may be implemented as a illuminance meter located on or around the dashboard or front windshield of the vehicle 100 to measure the ambient luminance at a location in front of the driver of the vehicle 100. When the current ambient luminance is greater than or equal to a threshold value, the control module 104 commands, signals, or otherwise instructs the imaging system 102 to configure the camera or other imaging device to obtain a high frame rate operating mode (e.g., 30 frames per second). On the other hand, when the current ambient illumination is less than the threshold, the control module 104 commands, signals, or otherwise instructs the imaging system 102 to configure the camera or other imaging device to obtain a low frame rate operating mode (e.g., 15 frames per second) to facilitate increasing the number of exposures.

[0025] After initializing the frame rate operating mode of the imaging system 102, the adaptive camera display process 200 analyzes the image data captured by the imaging system 102 to determine an estimated ambient illumination of the surrounding environment based on the image data and the current vehicle dynamics, and dynamically adjusts the frame rate operating mode of the imaging system 102 in response to changes in the estimated ambient illumination. In an exemplary implementation, the adaptive camera display process 200 receives or otherwise obtains current vehicle state information at 204, and then calculates or otherwise determines a representative exposure setting for a fused surround view image or other fusion of image data from various cameras or imaging devices based on the current vehicle state at 206. In this regard, the adaptive camera display process 200 utilizes the current state of the vehicle transmission and the current steering wheel angle associated with the vehicle steering system to determine which camera(s) are most relevant to the current operation of the vehicle 100, and then calculates the representative exposure setting as a weighted average of the exposure settings of the various cameras, wherein the weighting factors are dynamically determined based on the steering wheel angle and the transmission state to preferentially weight the cameras most relevant to the current vehicle operation. In this regard, the representative exposure setting (e D ) can be obtained from equation e D =∑i w i e i Indicates that, e i represents the exposure setting of the corresponding camera (i), w i represents the weighting factor assigned to the corresponding camera, where ∑ i w i =1.

[0026] For example, in an implementation where the imaging system 102 includes four cameras (e.g., a forward-looking, rear-looking, and two side-looking cameras), when the transmission of the vehicle 100 is in drive, indicating that forward operation of the vehicle 100 is most relevant, the weighting factors associated with the corresponding cameras may be controlled by the following set of equations:

[0027]

[0028]

[0029] as well as

[0030]

[0031] Where SWA represents the current steering wheel angle, and the weighting factor value is normalized so that according to the equation The sum of the corresponding weighting factors is equal to 1. In this regard, the adaptive camera display process 200 preferentially weights the exposure setting of the forward camera when the transmission is in drive, and gradually weights one of the left and right cameras according to the steering wheel angle according to the magnitude of the steering wheel angle, while de-emphasizing the exposure setting of the rearward camera and the other side camera opposite the steering wheel angle based on the magnitude of the angle. In a similar manner, when the transmission of the vehicle 100 is in reverse gear, indicating that the rearward operation of the vehicle 100 is most relevant, the weighting factors associated with the corresponding cameras can be controlled by the following set of equations:

[0032]

[0033]

[0034] as well as

[0035]

[0036] After determining the representative exposure setting, the adaptive camera display process 200 identifies or otherwise determines one or more regions of interest within the image data captured by the imaging system 102 at 208, and then uses the representative exposure setting to calculate or otherwise determine an estimated illuminance of the surrounding environment based on the image data corresponding to the identified regions of interest at 210. In an exemplary implementation, the control module 104 filters the image data from the different cameras of the imaging system 102 to exclude or otherwise remove subsets or regions of image data corresponding to the sky, shadows, or other visual artifacts or distortions (e.g., dark areas at the periphery of the camera's field of view due to the focal length of the camera lens (such as a fisheye lens)) that are not representative of the illuminance of the surrounding environment of the vehicle 100. After filtering out the subset of image data corresponding to the non-ambient areas, in an exemplary implementation, the remaining subset of image data corresponding to the ambient area of ​​interest is converted from RGB color model component values ​​to corresponding YUV color model component values ​​before calculating or otherwise determining the average value of the Y channel luminance component value of the identified ambient area of ​​interest. In this regard, the average value of the Y channel brightness component provides a robust average brightness measure of the vehicle environment for applications such as camera system auto-exposure / auto-gain algorithms, adaptive display right brightness algorithms, headlight / taillight brightness adjustment algorithms, etc.

[0037] In an exemplary implementation, the adaptive camera display process 200 utilizes the estimated illumination of the surrounding environment based on the image data to dynamically determine whether to adjust the frame rate setting of the imaging system 102 based on changes in the surrounding environment and different vehicle dynamics at 212, and when the estimated illumination indicates that the frame rate should be changed, command, signal, instruct, or otherwise configure the imaging system 102 to obtain a different frame rate at 214. In this regard, when the current frame rate setting of the imaging system 102 corresponds to a low frame rate operating mode, the control module 104 may automatically determine to adjust the frame rate setting of the imaging system 102 to a high frame rate operating mode when the estimated illumination of the image of the surrounding environment is greater than a threshold value, and provide corresponding commands or instructions to the imaging system 102 to configure the camera of the imaging system 102 to obtain the high frame rate operating mode. Similarly, when the current frame rate setting of the imaging system 102 corresponds to the high frame rate operation mode, the control module 104 may automatically determine to adjust the frame rate setting of the imaging system 102 to the low frame rate operation mode when the estimated illumination of the image of the surrounding environment is below the threshold, and provide a corresponding command or instruction to the imaging system 102 to configure the camera of the imaging system 102 to obtain the low frame rate operation mode. In this regard, the threshold for dynamically and adaptively switching the frame rate may include an offset to provide hysteresis for preventing the imaging system 102 from continuously switching back and forth between different frame rates.

[0038] Still reference Figure 2 In an exemplary implementation, when the adaptive camera display process 200 changes the frame rate of the imaging system 102, at 216, the adaptive camera display process 200 temporarily smooths the brightness of subsequent image data captured using different frame rates on consecutive frames within the transition period. In this regard, changes in camera frame rates can cause sudden changes in image brightness, which can affect the ability of a driver or other user to accurately perceive the surrounding environment shown on the display device 106. Therefore, in an exemplary implementation, the control module 104 gradually increases or decreases the image brightness in response to changing the frame rate setting of the imaging system 102 for the duration of the transition period before stopping any smoothing or adjustment of the image brightness. In one or more implementations, the control module 104 linearly smooths the perceived image brightness when the frame rate transitions using a multiplier that increases or decreases in a linear or semi-linear manner during the transition period.

[0039] For example, in one implementation, when switching from a high frame rate operating mode to a low frame rate operating mode, a smoothing multiplier (δ t ) can be obtained from the equation δ t =δ0+(t-1)δ step to control, where t represents the corresponding frame number of the corresponding frame of the low frame operation mode until the maximum number of transition frames (t w ) until δ step represents the amount by which the value of the smoothing multiplier is incrementally adjusted, the smoothing multiplier being given by the equation , and δ0 is a constant based on the relationship or ratio between the brightness of the previous image data captured in the high frame rate operation mode and the brightness of the initial image data when switching to the low frame rate operation mode. Conversely, when switching from the low frame rate operation mode to the high frame rate operation mode, the smoothing multiplier (δ t ) can be obtained from the equation δ t =δ0-(t-1)δ step to control, where t represents the corresponding frame number of the corresponding frame of the low frame operation mode until the maximum number of transition frames (t w ), δ step represents the amount by which the value of the smoothing multiplier is incrementally adjusted, the smoothing multiplier being given by the equation to control, and δ0 is equal to 1.

[0040] Reference again Figure 2In an exemplary implementation, at 218, the adaptive camera display process 200 also uses the current vehicle state information to dynamically adapt and adjust the display brightness in substantially real time based on the estimated illumination of the current surrounding environment. In this regard, the control module 104 calculates or otherwise determines the brightness setting value based on the estimated illumination derived from the region of interest in the image data (determined at 210) and a weighted average of the current brightness measurements output by the flow meter or other illumination sensing element 108, wherein the weighting factors vary depending on the current transmission state of the vehicle 100. For example, in one implementation, the adaptive display brightness setting value for the current frame is determined to be the original brightness setting value calculated for the current frame (b t ) and the original brightness setting value calculated for the previous frame (b t-1 ), which can be expressed by the equation to control, where α is a weighting factor with a value between 0 and 1. It can be controlled according to the equation Fusion illumination estimation based on current frame To calculate the original brightness setting value, where b min Indicates the minimum original brightness setting value calculated for the previous frame, b max Represents the maximum original brightness setting value calculated for the previous frame. represents the maximum estimated illuminance derived from a frame of image data, g is a calibratable parameter that controls the curvature of change of the adaptive display brightness setting value, and the fused illuminance estimate is calculated based on the estimated illuminance derived from the current image data frame and the current brightness measurement value.

[0041] In one or more implementations, the fused illumination estimate of the current frame According to the equation The mode that varies according to the current transmission state is calculated as a weighted average of the estimated illumination derived from the current frame of image data and the current brightness measurement, where μ c is the estimated illuminance derived from the current frame of image data, l is the current brightness measurement, and The fused illuminance estimate may be calculated based on a ratio of a current luminance measurement to a threshold luminance value (e.g., 65,000 lumens). In this regard, the fused illuminance estimate may be gradually weighted with an estimated illuminance derived from a current frame of image data in a low light environment or when the vehicle 100 is in reverse, parked, or otherwise not traveling in a forward direction corresponding to the forward position of the illuminance sensing element 108.

[0042] In one or more exemplary implementations, in order to adaptively adjust the display brightness, according to the equation μ c =∑ i w i μi , the estimated illumination derived from the current frame of image data is calculated as a weighted sum of the estimated illumination derived from the corresponding image data from each camera of the imaging system 102, where μ i represents the estimated illumination associated with the corresponding camera, w i represents the weighting factor associated with the corresponding camera. In an exemplary implementation, the weighting factor satisfies the equation ∑ i w i =1, and w i According to the equation w i =γ i β i varies according to the relationship between the camera's orientation or field of view and the current transmission state of the vehicle 100, where γ i represents the weighting factor based on the transmission, β i represents a weighting factor corresponding to the viewing angle of the corresponding camera relative to the road. For example, when the current transmission state corresponds to drive or another forward gear, the adaptive camera display process 200 can adjust the transmission-based weighting factor to increase the weight of the front camera and decrease the transmission-based weight of the rear camera, and conversely, when the current transmission state corresponds to reverse gear, the adaptive camera display process 200 can adjust the transmission-based weighting factor to increase the weight of the rear camera and decrease the transmission-based weight of the front camera. The adaptive camera display process 200 can also use the direction of travel, navigation map data, and other information that may be obtained from other vehicle systems 110 to calculate, estimate, or otherwise determine the road viewing angle of a particular camera to incrementally or incrementally adjust the viewing angle weighting factor β based on the direction of travel of the vehicle 100 and the orientation of the corresponding camera relative to the direction of travel or the orientation of the current lane of travel. i .

[0043] Still reference Figure 1-Figure 2 After determining the adaptive display brightness setting value, the control module 104 automatically commands, signals, or otherwise instructs the display device 106 to increase or decrease the brightness of the display device 106 accordingly to adapt to the current illumination of the vehicle 100 and the surrounding environment in a manner that takes into account the current transmission state and direction of travel of the vehicle 100. By using a moving average to determine the adaptive display brightness setting value over consecutive frames, the display brightness transitions smoothly, thereby improving the perception of the GUI display and the related images depicted on the display device 106.

[0044] The loop defined by 204, 206, 208, 210, 212, 214, 216, and 218 can be infinitely repeated during the operation of vehicle 100 to dynamically adjust one or more of the frame rate, image brightness, and display brightness substantially in real time in response to changes in the illuminance associated with the surrounding environment and / or changes in the vehicle state, so as to improve the driver's perception of the image depicted on display device 106. By smoothly or gradually adjusting the brightness associated with the image data and / or the brightness setting of display device 106, the human factor is improved by avoiding sudden changes in brightness or perceived brightness, where sudden changes in brightness or perceived brightness can affect the driver's ability to determine or evaluate the surrounding environment depicted on display device 106.

[0045] Figure 3 An exemplary implementation of the frame rate selection process 300 is described, where the frame rate selection process 300 is adapted to be implemented in conjunction with the adaptive camera display process 200 (e.g., at 212 and 214) to automatically change the frame rate substantially in real time based on the estimated or perceived brightness of the surrounding environment, thereby dynamically and adaptively responding to changes in ambient illuminance and / or vehicle state. For purposes of explanation, the following description may refer to the elements described above in conjunction with Figure 1 the elements mentioned. Although parts of the frame rate selection process 300 may be performed by different elements of the vehicle system, for purposes of explanation, the subject matter may be mainly described in the context where the frame rate selection process 300 is mainly performed by the control module 104.

[0046] The frame rate selection process 300 begins by determining at 302 whether to configure the imaging system 102 to obtain a higher frame rate operation mode or a lower frame rate operation mode based on the current brightness measured by the illuminance sensing element 108. In this regard, when the current value of the measured brightness is greater than or equal to the initial selection threshold, at 304, the frame rate selection process 300 automatically configures the imaging system 102 to obtain a higher frame rate operation mode. On the other hand, when the current value of the measured brightness is less than the initial selection threshold (e.g., l < l0), at 314, the frame rate selection process 300 automatically configures the imaging system 102 to obtain a lower frame rate operation mode.

[0047] After configuring the imaging system 102 to obtain a high frame rate operation mode, the frame rate selection process 300 calculates or otherwise determines an estimated illuminance of the surrounding environment based on the image data (e.g., at 210), then at 306 verifies that the estimated illuminance of the captured image data based on the region of interest is higher than a lower brightness threshold for switching the frame rate, and automatically switches from the high frame rate operation mode to the low frame rate operation mode when the estimated illuminance is lower than the lower brightness threshold (e.g., l < l1). In this regard, the lower brightness threshold for switching from the high frame rate operation mode to the low frame rate operation mode can be less than the initially selected threshold (e.g., l1 < l0) to prevent spurious transitions or switches between the frame rate operation modes. In this way, when the estimated illuminance derived from the captured image data indicates that the perceived ambient illumination of the surrounding environment has decreased sufficiently, the control module 104 automatically converts the imaging system 102 from the high frame rate operation mode to the low frame rate operation mode.

[0048] At 308, the frame rate selection process 300 also verifies that the estimated illuminance is higher than an intermediate brightness threshold, and when the estimated illuminance is less than the intermediate brightness threshold (e.g., l < l2), at 310, the frame rate selection process 300 determines whether the perceived brightness is attributable to the vehicle brake lights. In this regard, the intermediate brightness threshold can be greater than the lower threshold for automatically switching from the high frame rate operation mode to the low frame rate operation mode (e.g., l1 < l2), where the frame rate selection process 300 analyzes the current transmission state of the vehicle 100 and the amount of red in the captured image data to verify that the difference between the estimated illuminance and the lower brightness threshold is not due solely to the vehicle brake lights. For example, in one implementation, the frame rate selection process 300 calculates or otherwise determines a red fraction of the captured image data (e.g., the percentage of the red component relative to the other RGB components), and then, when the current transmission gear state corresponds to reverse and the red fraction is greater than a threshold percentage (e.g., 30%), determines that the estimated illuminance is attributable to the vehicle brake lights. In this regard, when the estimated illuminance derived from the captured image data is relatively low and is attributable to the vehicle brake lights rather than the ambient illumination of the surrounding environment, the control module 104 automatically converts the imaging system 102 from the high frame rate operation mode to the low frame rate operation mode.

[0049] Furthermore, in an exemplary implementation, when the estimated illuminance is less than the intermediate brightness threshold and is not attributable to vehicle brake lights, the frame rate selection process 300 also verifies or otherwise confirms that the brightness of each respective image frame captured by the respective camera or imaging device of the imaging system 102 is greater than the brightness threshold at 312. In this regard, if the brightness of one of the forward, rearward, or side camera images is below the brightness threshold, the frame rate selection process 300 may automatically transition the imaging system 102 from a higher frame rate operating mode to a lower frame rate operating mode to increase the likelihood that each of the images captured from the respective camera of the imaging system 102 has sufficient brightness to be presented on the display device 106.

[0050] When the estimated illuminance of the captured image data remains above the lower brightness threshold and the intermediate brightness threshold, the frame rate selection process 300 maintains the higher frame rate mode of operation (e.g., at 212), and allows the adaptive camera display process 200 to dynamically adjust the display brightness based on the estimated illuminance of the current surrounding environment perceived in the higher frame rate mode of operation at 218. On the other hand, when the estimated illuminance of the captured image data is below the lower brightness threshold at 306, one of the images captured at 312 is below the threshold brightness, or the estimated illuminance of the captured image data is attributed to vehicle brake lights, the frame rate selection process 300 automatically switches the imaging system 102 from the higher frame rate mode of operation to the lower frame rate mode of operation (e.g., at 214 of the adaptive camera display process 200) at 314, which initiates a transition period, wherein within the transition period, the adaptive camera display process 200 smoothes the perceived image brightness of the image data subsequently captured in the lower frame rate mode of operation at 216. Thus, when the frame rate selection process 300 automatically changes the frame rate operating mode from a higher frame rate operating mode to a lower frame rate operating mode, the adaptive camera display process 200 smoothes the perceived image brightness in order to avoid abrupt changes in brightness that may occur due to the increased exposure time associated with the lower frame rate.

[0051] Still reference Figure 3, in a manner similar to that described in the context of the higher frame rate operating mode, after configuring the imaging system 102 to obtain the lower frame rate operating mode, the frame rate selection process 300 calculates or otherwise determines an estimated illuminance of the surrounding environment based on the image data (e.g., at 210), and then verifies at 316 that the estimated illuminance based on the captured image data of the region of interest does not exceed the upper brightness threshold for switching frame rates. In this regard, the upper brightness threshold is greater than the intermediate brightness threshold by an offset (e.g., Δl) to provide hysteresis to prevent switching between different frame rates. When the estimated illuminance is greater than the upper brightness threshold (e.g., l>l2+Δl), the frame rate selection process 300 automatically switches the imaging system 102 from the lower frame rate operating mode to the higher frame rate operating mode.

[0052] Additionally, in an exemplary implementation, the frame rate selection process 300 also verifies or otherwise confirms whether the brightness of each respective image frame captured by the respective camera or imaging device of the imaging system 102 is greater than a brightness threshold at 318. In this regard, when the brightness of each of the front, rear, or side camera images is above the brightness threshold, the frame rate selection process 300 may automatically transition the imaging system 102 from a lower frame rate operating mode to a higher frame rate operating mode to improve visibility of the captured images.

[0053] As described above, when the frame rate selection process 300 automatically switches the imaging system 102 from the lower frame rate operating mode to the higher frame rate operating mode at 214, a transition period begins, wherein during the transition period, the adaptive camera display process 200 smoothes the perceived image brightness of image data subsequently captured in the higher frame rate operating mode at 216. Thus, when the frame rate selection process 300 automatically changes the lower frame rate operating mode to the higher frame rate operating mode, the adaptive camera display process 200 smoothes the perceived image brightness to avoid a sudden dimming that may occur due to the reduced exposure time associated with the higher frame rate.

[0054] By automatically and adaptively switching between different frame rate operating modes, and smoothing the resulting image data accordingly, the vehicle 100 can move from a bright environment to a dark environment, and vice versa, without degrading the driver's ability to perceive the GUI displayed on the display device 106. In addition, the brightness or brightness of the captured image data and the corresponding GUI depicted on the display device 106 can adapt to the perceived environment in a manner that reflects the current operation of the vehicle 100 (e.g., current transmission gear state, current steering wheel angle, current vehicle direction of travel, etc.) and the ambient illumination of the area of ​​interest in the surrounding environment. For example, surround view camera systems and fused images can play an important role in parking assistance and other viewing and perception applications, but are susceptible to changes in lighting due to physical phenomena (e.g., dusk or dawn) or when the vehicle enters or exists in a different environment (e.g., a garage). Due to the subject matter described herein, the surround view camera system can quickly adapt to these types of changes while avoiding sudden changes in display brightness that can reduce perception and traffic safety. Using steering wheel angle, transmission state, and potentially other factors, the subject matter described herein is able to reliably determine the brightness around the host vehicle by local analysis limited to those areas that are most likely to be of interest given the current vehicle state, and then adapt the captured image to the current environment using adaptive camera frame rate and camera exposure techniques while smoothing brightness transitions on the display. This improves the driver's visibility of the surrounding environment and improves perception and eye comfort by quickly adjusting image and display brightness in response to lighting changes, thereby improving traffic safety, which in turn improves driver confidence and situational awareness and allows the driver to see multiple views and image frames in a more coherent manner.

[0055] For the sake of brevity, conventional techniques related to lane keeping assist, driver assistance features, autonomous vehicles, and other functional aspects of the system (and the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in implementations of the present subject matter.

[0056] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Thus, any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. All implementations described herein are exemplary implementations, which are provided to enable those skilled in the art to make or use the invention, and are not intended to limit the scope of the invention, which is defined by the claims.

[0057] Those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the implementation disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. Some implementations are described above with respect to functions and / or logic block components (or modules) and various processing steps. However, it should be understood that such block components (or modules) can be implemented by any number of hardware, software and / or firmware components configured to perform a specified function. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention.

[0058] When implemented in software or firmware, the various elements of the system described herein are essentially code segments or instructions that perform various tasks. A program or code segment can be stored in a processor-readable medium, or transmitted on a transmission medium or communication path by a computer data signal contained in a carrier wave. "Computer-readable medium", "processor-readable medium" or "machine-readable medium" may include any medium that can store or transmit information. Examples of processor-readable media include electronic circuits, semiconductor storage devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Computer data signals may include any signal that can be propagated on a transmission medium such as an electronic network channel, optical fiber, air, electromagnetic path or RF link, etc. Code segments may be downloaded via computer networks such as the Internet, intranets, LANs, etc.

[0059] In this document, relational terms such as first and second may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Ordinal numbers such as "first", "second", "third", etc. simply represent different singles among a plurality and do not imply any order or sequence unless specifically defined by the claim language. The order of the text in any claim does not mean that the process steps must be performed in a temporal or logical order according to such order unless specifically defined by the claim language. Without departing from the scope of the present invention, the process steps may be interchanged in any order as long as such interchange does not contradict the claim language and is logically consistent.

[0060] In addition, the foregoing description may relate to "coupled" elements or nodes or features. As used herein, unless otherwise expressly stated, "coupling" illustrates that an element / node / feature is directly or indirectly connected to another element / node / feature (or directly or indirectly communicates with it), and is not necessarily mechanically connected. For example, two elements can be coupled to each other physically, electronically, logically or in any other way by one or more additional elements. Therefore, although the accompanying drawings may show an exemplary arrangement of elements directly connected to each other, additional intermediate elements, equipment, features or parts may be present in the implementation of the illustrated subject matter. In addition, a certain term may also be used herein only for reference purposes, and is therefore not intended to be limited.

[0061] Although at least one exemplary aspect has been presented in the foregoing detailed description, it should be understood that there are a large number of variations. It should also be understood that one or more exemplary aspects are merely examples and are not intended to limit the scope, applicability, or configuration of the present disclosure in any way. On the contrary, the foregoing detailed description will provide a convenient roadmap for implementing one or more exemplary aspects for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present disclosure as set forth in the attached claims and their legal equivalents.

Claims

1. A method of assisting operation of a vehicle, the method comprising: obtaining, by a control module associated with the vehicle, image data from an imaging system on the vehicle, the image data being captured at the imaging system using an initial frame rate; obtaining, by the control module, current vehicle status information associated with the vehicle; determining, by the control module, a representative exposure setting for the imaging system based at least in part on the current vehicle state information; determining, by the control module, an estimated illumination of a surrounding environment of the vehicle based at least in part on the image data and the representative exposure setting; automatically configuring, by the control module, the imaging system to obtain a second frame rate different from the initial frame rate based on the estimated illumination; as well as Then: obtaining, from the imaging system, subsequent image data captured using the second frame rate; providing a graphical user interface including a graphical representation of the subsequent image data on a display device associated with the vehicle; as well as After automatically configuring the imaging system to obtain the second frame rate, display characteristics of the graphical user interface are adjusted for a temporary period of time.

2. The method according to claim 1, wherein: Adjusting the display characteristic includes gradually adjusting the brightness of the graphical representation of the subsequent image data over the temporary period of time.

3. The method according to claim 2, wherein: Gradually adjusting the brightness includes configuring the brightness of the graphical representation of the initial frame of the subsequent image data based on a weighted average of the initial frame of the subsequent image data captured using the second frame rate and a previous frame of the image data captured using the initial frame rate.

4. The method of claim 1 , further comprising identifying, by the control module, one or more regions of interest within the image data, wherein: Determining the estimated illumination includes: Converting the RGB component values ​​of the one or more regions of interest into YUV component values; and The Y-channel brightness component values ​​of the YUV component values ​​over the one or more regions of interest are averaged.

5. The method according to claim 4, wherein: Determining the representative exposure setting includes calculating the representative exposure setting as a weighted sum of respective exposure settings associated with respective imaging devices of the imaging system, wherein respective weighting factors associated with respective imaging devices are affected by a current state of a transmission system associated with the vehicle.

6. The method according to claim 1, wherein the current vehicle status information includes a current status of a transmission system of the vehicle, wherein: Determining the representative exposure setting includes calculating the representative exposure setting as a weighted sum of respective exposure settings associated with respective imaging devices of the imaging system, wherein respective weighting factors associated with respective imaging devices are affected by the current state of the transmission system associated with the vehicle.

7. The method according to claim 1, wherein: Adjusting the display characteristic includes gradually increasing the brightness of the graphical representation of the subsequent image data over the temporary period of time after automatically configuring the imaging system to obtain a lower frame rate.

8. The method according to claim 1, wherein: Adjusting the display characteristic includes gradually reducing the brightness of the graphical representation of the subsequent image data over the temporary period of time after automatically configuring the imaging system to obtain a higher frame rate.

9. A vehicle system comprising: an illuminance sensing element for providing a measured brightness; Steering system; Display devices; An imaging system, including a plurality of imaging devices; as well as a control module coupled to the illumination sensing element, the steering system, the display device, and the imaging system, wherein the control module can be configured to: configuring the imaging system to obtain an initial frame rate based on the measured brightness; determining a representative exposure setting for the imaging system based at least in part on a current steering angle associated with the steering system; determining an estimated illumination of an ambient environment based at least in part on the representative exposure setting and image data from the imaging system captured using the initial frame rate; automatically configuring the imaging system to obtain a second frame rate different from the initial frame rate based on the estimated illumination; obtaining, from the imaging system, subsequent image data captured using the second frame rate; providing, on the display device, a graphical user interface including a graphical representation of the subsequent image data; and After automatically configuring the imaging system to obtain the second frame rate, display characteristics of the graphical user interface are adjusted for a temporary period of time.

10. The vehicle system of claim 9, further comprising a transmission system, wherein: The control module can be configured to determine the representative exposure setting in a manner that is influenced by a current state of the transmission system.

Images