A visual monitoring-based adaptive dimming system and method for a vehicle cabin

Abstract

The application provides a kind of based on visual monitoring's car cabin adaptive light adjusting system and method, it can be according to the brightness condition of current environment and improve the image quality collected by light supplement, and can realize the brightness of vehicle-mounted display screen in cabin according to the brightness condition of current environment adaptive adjustment, improve driving experience, system includes first, second, third in-vehicle monitoring camera and driver monitoring camera;According to the brightness condition of current environment from the image data of driver visual monitoring device;Car cabin monitoring controller judges the brightness condition of current environment with the set light dim threshold, informs driver visual monitoring device to open infrared light supplement lamp according to the result of judgment;Car cabin monitoring controller sends the brightness condition of current environment to vehicle-mounted display screen;Vehicle-mounted display screen adjusts brightness according to the brightness condition of current environment received.

Description

Technical Field

[0001] This invention relates to the field of cockpit monitoring technology, and specifically to an adaptive dimming system and method for automotive cockpits based on visual monitoring. Background Technology

[0002] Cockpit vision monitoring systems rely on camera image input and utilize deep learning algorithms to monitor drivers and passengers in real time for facial recognition, fatigue monitoring, attention monitoring, and behavioral action monitoring. Image quality is therefore a crucial aspect of the entire cockpit vision monitoring system. In normal driving environments, there are situations where lighting is relatively low. To ensure the quality of the captured images, infrared lights are needed for supplemental lighting. How to accurately and in real-time obtain the brightness of the vehicle's cabin under the current environment is a problem that urgently needs to be solved. Solving this problem without adding additional light sensors has great application potential. As the level of automotive intelligence increases, the number of displays in cars is also increasing, and there is a need for adaptive brightness of these displays in different environments. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides an adaptive dimming system and method for automotive cabins based on visual monitoring. This system can supplement lighting to improve the quality of acquired images according to the current ambient brightness, and can adaptively adjust the brightness of the in-vehicle display screen in the cabin according to the current ambient brightness, thereby improving the driving experience.

[0004] The technical solution is as follows: A visual monitoring-based adaptive dimming system for automotive cabins, characterized in that it includes: a driver visual monitoring device, an in-vehicle display screen, and an automotive cabin monitoring controller, characterized in that:

[0005] The driver visual monitoring device includes: a first in-vehicle monitoring camera, a second in-vehicle monitoring camera, a third in-vehicle monitoring camera, and a driver monitoring camera. The first in-vehicle monitoring camera, the second in-vehicle monitoring camera, and the third in-vehicle monitoring camera are IR-RGB cameras, and the driver monitoring camera is an IR camera.

[0006] The vehicle cabin monitoring controller receives images collected by the driver's visual monitoring device to monitor the driver and passengers. The vehicle cabin monitoring controller obtains the current ambient brightness based on the image data from the driver's visual monitoring device. The vehicle cabin monitoring controller compares the current ambient brightness with a set dim light threshold and, based on the comparison result, notifies the driver's visual monitoring device to turn on the infrared auxiliary light. The vehicle cabin monitoring controller sends the current ambient brightness to the in-vehicle display screen. The in-vehicle display screen adjusts its brightness according to the received current ambient brightness.

[0007] Furthermore, the driver monitoring camera is installed on the A-pillar on the driver's side inside the vehicle; the first in-vehicle monitoring camera is installed on the rearview mirror inside the vehicle; the second in-vehicle monitoring camera is installed on the B-pillar on the driver's side inside the vehicle; and the third in-vehicle monitoring camera is installed on the B-pillar on the passenger side inside the vehicle.

[0008] Furthermore, the in-vehicle display screen includes an instrument panel screen, a central control screen, and a headrest screen.

[0009] Furthermore, the first in-vehicle monitoring camera, the second in-vehicle monitoring camera, the third in-vehicle monitoring camera, and the driver monitoring camera of the driver vision monitoring device share an ISP chip integrated on the SOC chip of the vehicle cockpit monitoring controller for image processing.

[0010] Furthermore, the first, second, and third in-vehicle monitoring cameras are used to detect the occupant area. The vehicle cabin monitoring controller performs image recognition on the occupant's behavior and actions in the image data collected by the first, second, and third in-vehicle monitoring cameras, and links vehicle control functions based on the recognition results. The vehicle cabin monitoring controller also performs image recognition on the driver in the image data collected by the driver monitoring camera, performing facial recognition, fatigue monitoring, and attention monitoring.

[0011] A visual monitoring-based adaptive dimming method for automotive cabins, characterized by being implemented based on the aforementioned visual monitoring-based adaptive dimming system for automotive cabins, includes the following steps:

[0012] The driver's visual monitoring device collects image data and sends it to the vehicle's cockpit monitoring controller;

[0013] The vehicle cabin monitoring controller obtains the current ambient brightness based on image data from the driver's visual monitoring device;

[0014] The vehicle cabin monitoring controller judges the current ambient brightness against the set dim light threshold, and notifies the driver's visual monitoring device to turn on the infrared auxiliary light based on the judgment result.

[0015] The vehicle cabin monitoring controller sends the current ambient brightness information to the in-vehicle display screen;

[0016] The in-vehicle display adjusts its brightness based on the received ambient light level.

[0017] Furthermore, under different light intensities, images are captured from the camera, and the RAW data of the images is statistically analyzed to calculate the average values ​​of the G and IR components of the image. The average values ​​of the G and IR components are used to represent the gray-scale mean of the G and IR components of the image, respectively, thus obtaining the mapping relationship between the gray-scale mean of the image data and the light intensity. Based on the mapping relationship, different brightness levels are obtained. The vehicle cabin monitoring controller sends the brightness level of the captured image to the vehicle display screen, and the brightness of the vehicle display screen is adaptively adjusted according to the brightness level.

[0018] Furthermore, the current ambient brightness is compared with the set dimness threshold as follows:

[0019] The vehicle cabin monitoring controller receives RAW data collected by the camera of the driver's vision monitoring device, performs statistical analysis on each frame of RAW data, and calculates the average value of the G component and IR component of the image. The average value of the G component and IR component is used to represent the grayscale mean of the G component and IR component of the image, respectively.

[0020] When the average value of G is less than the average value of IR, the light is considered dim, and the infrared supplementary light is turned on.

[0021] When the average value of G is greater than the average value of IR, the difference between the G component and the IR component is calculated and combined with the camera's gain parameter and shutter parameter to obtain a brightness estimate. When the brightness estimate is less than the dim light threshold, it is considered that the light is dim and the infrared fill light is turned on; when the brightness estimate is greater than the dim light threshold, it is considered that the light is normal and the infrared fill light is turned off.

[0022] Furthermore, when the driver monitoring camera activates its infrared fill light for infrared exposure, the first, second, and third in-vehicle monitoring cameras are not allowed to perform exposure.

[0023] A computer-readable storage medium is characterized in that: the computer-readable storage medium is used to store a program for executing the above-described adaptive dimming method for automotive cabins based on visual monitoring.

[0024] This invention provides a visual monitoring-based adaptive dimming system and method for automotive cockpits. It utilizes image data collected by a camera in a driver's visual monitoring system to determine the current ambient brightness, thereby controlling the activation and deactivation of the camera's infrared lights to ensure image quality. The cockpit visual monitoring system can better perform monitoring functions such as facial recognition, fatigue detection, attention monitoring, and behavioral action monitoring. This invention saves on camera costs, as the camera does not require additional light-sensing components. Furthermore, it establishes a mapping relationship between camera image data and light intensity, classifying different brightness levels based on this relationship. This can be output through the automotive cockpit monitoring controller and used as a basis for adaptive brightness control of screens such as the central control screen, instrument panel screen, and headrest screens, improving the driving and riding experience for drivers and passengers. Attached Figure Description

[0025] Figure 1 This is a block diagram of the automotive cabin adaptive dimming system based on visual monitoring in the embodiment.

[0026] Figure 2 This is a schematic diagram showing the camera placement positions in the visual monitoring-based adaptive dimming system for automotive cabins in this embodiment.

[0027] Figure 3 This is a schematic diagram illustrating the steps of the adaptive dimming method for a car cabin based on visual monitoring in the embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] As shown in the background art, in order to ensure the image quality acquired by the cockpit visual monitoring system, it is of great practical significance to accurately and in real time obtain the brightness of the vehicle cockpit under the current environment without adding additional light sensors, and to realize adaptive adjustment of the brightness of the vehicle display screen under different environments.

[0030] In response to this, see Figure 1 The present invention provides an adaptive dimming system for a car cabin based on visual monitoring, including: a driver visual monitoring device, an in-vehicle display screen and a car cabin monitoring controller 300, wherein the in-vehicle display screen includes an instrument panel screen 201, a central control screen 202 and a headrest screen 203.

[0031] The driver visual monitoring device includes: a first in-vehicle monitoring camera 101, a second in-vehicle monitoring camera 102, a third in-vehicle monitoring camera 103, and a driver monitoring camera 104. The first in-vehicle monitoring camera 101, the second in-vehicle monitoring camera 102, and the third in-vehicle monitoring camera 103 are IR-RGB cameras, while the driver monitoring camera 104 is an IR camera. The first in-vehicle monitoring camera 101, the second in-vehicle monitoring camera 102, and the third in-vehicle monitoring camera 103 support both IR and RGB modes, while the driver monitoring camera 104 supports IR mode. IR mode represents a grayscale image, and RGB mode represents a normal color image.

[0032] See Figure 2 The driver vision monitoring device is equipped with four cameras inside the vehicle. The driver monitoring camera 104 is installed on the A-pillar on the driver's side, the first in-vehicle monitoring camera 101 is installed on the rearview mirror, the second in-vehicle monitoring camera 102 is installed on the B-pillar on the driver's side, and the third in-vehicle monitoring camera 103 is installed on the B-pillar on the passenger side. By setting the installation positions of the first in-vehicle monitoring camera 101, the second in-vehicle monitoring camera 102, the third in-vehicle monitoring camera 103, and the driver monitoring camera 104 in this embodiment, images of the occupants in the cabin can be accurately obtained. Compared with the traditional single-camera solution, the image data obtained is more comprehensive. The vehicle cabin monitoring controller, the instrument panel, the central control screen, and the headrest screen of the vehicle display are adapted accordingly according to the actual vehicle design. The first in-vehicle monitoring camera 101, the second in-vehicle monitoring camera 102, and the third in-vehicle monitoring camera 103 can be used to detect the occupant area, perform image recognition on some of the occupant's behaviors and actions, and then link the vehicle control functions, such as: when an occupant smokes, the car window is opened automatically; when monitoring the occupant's gestures, gestures are used to change songs, like, take photos, etc.; when an occupant makes a phone call, the car window is closed automatically and the air conditioning fan speed is reduced.

[0033] The vehicle cabin monitoring controller receives images captured by the driver's visual monitoring device and monitors the driver and passengers based on deep learning algorithms, including facial recognition, fatigue monitoring, attention monitoring, and behavioral monitoring. Simultaneously, the controller obtains the current ambient brightness based on image data from the camera. It then compares this ambient brightness with a set dimness threshold and, based on the result, instructs the driver's visual monitoring device to activate the infrared auxiliary light. Finally, the controller sends the ambient brightness information to the in-vehicle display screen, which adjusts its brightness accordingly.

[0034] In one embodiment of the present invention, the first in-vehicle monitoring camera 101, the second in-vehicle monitoring camera 102, the third in-vehicle monitoring camera 103 and the driver monitoring camera 104 of the driver vision monitoring device share an ISP chip integrated on the SOC chip of the vehicle cockpit monitoring controller for image processing. The four cameras share an ISP chip integrated on the SOC chip of the controller for image processing, which saves the ISP chip of the camera and reduces the cost.

[0035] See Figure 3 In an embodiment of the present invention, a visual monitoring-based adaptive dimming method for a car cabin is also provided, which is implemented based on the aforementioned visual monitoring-based adaptive dimming system for a car cabin, and specifically includes the following steps:

[0036] Step 1: The driver's visual monitoring device collects image data and sends it to the vehicle's cockpit monitoring controller;

[0037] Step 2: The vehicle cabin monitoring controller obtains the current ambient brightness based on image data from the driver's visual monitoring device;

[0038] Step 3: The vehicle cabin monitoring controller judges the current ambient brightness against the set dim light threshold, and notifies the driver's visual monitoring device to turn on the infrared auxiliary light based on the judgment result;

[0039] Step 4: The vehicle cabin monitoring controller sends the current ambient brightness information to the in-vehicle display screen;

[0040] Step 5: The vehicle display adjusts its brightness based on the received ambient light level.

[0041] In one embodiment, in step 1, the first in-vehicle monitoring camera 101, the second in-vehicle monitoring camera 102, and the third in-vehicle monitoring camera 103 of the driver visual monitoring device can be used to detect the occupant area, and the driver monitoring camera 104 can collect image data of the driver.

[0042] In step 2, the vehicle cabin monitoring controller receives RAW data collected by the camera of the driver vision monitoring device, performs statistical analysis on each frame of RAW data, and calculates the average value of the G component and IR component of the image. The average value of the G component and IR component is used to represent the grayscale mean of the G component and IR component of the image, respectively.

[0043] In step 3, the current ambient brightness is compared with the set dimness threshold as follows:

[0044] When the average value of G is less than the average value of IR, the light is considered dim, and the infrared supplementary light is turned on.

[0045] When the average G value is greater than the average IR value, the difference between the G and IR components is calculated and combined with the camera's gain and shutter parameters to obtain a brightness estimate. The larger the gain and shutter parameters are, the brighter the image. Different sensors have different processing methods. Basically, it is the average value after statistics, and the gain and shutter are divided.

[0046] When the estimated brightness is less than the dim light threshold, the light is considered dim and the infrared auxiliary light is turned on; when the estimated brightness is greater than the dim light threshold, the light is considered normal and the infrared auxiliary light is turned off. By logically controlling the infrared lights of the driver monitoring camera and the in-vehicle monitoring camera, better image quality is provided.

[0047] In step 4, the brightness of the in-vehicle monitoring camera can be calibrated and divided into different brightness levels. The specific calibration process involves acquiring images from the camera under different light intensities, statistically analyzing the RAW data of the images, and calculating the average values ​​of the G and IR components. These average values ​​represent the grayscale mean of the G and IR components, respectively, thus obtaining the mapping relationship between the grayscale mean of the image data and the light intensity. Based on this mapping relationship, different brightness levels are determined. The vehicle cabin monitoring controller sends the acquired image brightness levels to the instrument panel, central control screen, headrest screen, etc., via the CAN bus. The mapping relationship is based on the grayscale mean, outputting a mapping table, sampling different ambient light conditions, and statistically analyzing the average grayscale value range. This determines which range of grayscale values ​​corresponds to which ambient light intensity, resulting in the output brightness level.

[0048] In step 5, the brightness of the in-vehicle display screen is adaptively adjusted according to the brightness level to provide users with a better viewing experience. The calculated light level can be output by the controller and can be used as a basis for controlling the brightness of other screens in the car cabin.

[0049] In this embodiment, step 3 further involves exposure timing control for the driver monitoring camera and the in-vehicle monitoring camera. When the driver monitoring camera is performing infrared exposure, the first, second, and third in-vehicle monitoring cameras are not allowed to be exposed. This is because the infrared light from the driver monitoring camera affects the image from the in-vehicle monitoring cameras, causing color distortion and negatively impacting the recognition effect. Furthermore, when the user views the image from the in-vehicle monitoring cameras, the image appears color-distorted, resulting in a poor user experience. Similarly, when one of the in-vehicle monitoring cameras is in a low-light environment, the infrared light of one of the first, second, and third in-vehicle monitoring cameras is activated, entering IR mode, and the other two cameras simultaneously switch to IR mode. This method improves image quality, thereby enhancing the recognition effect of the monitoring.

[0050] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program for executing the above-described adaptive dimming method for automotive cabins based on visual monitoring.

[0051] Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory stores programs, and the processor executes these programs after receiving execution instructions.

[0052] A processor can be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor.

[0053] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] The embodiments of the present invention are described with reference to flowchart illustrations of methods or computer program products according to embodiments of the invention. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart.

[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in the flowchart.

[0056] The above provides a detailed description of the adaptive dimming method, system, and computer-readable storage medium for automotive cockpits based on visual monitoring provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A visual monitoring-based adaptive dimming system for automotive cabins, characterized in that, include: The driver visual monitoring device, the in-vehicle display screen, and the vehicle cabin monitoring controller are characterized by: The driver visual monitoring device includes: a first in-vehicle monitoring camera, a second in-vehicle monitoring camera, a third in-vehicle monitoring camera, and a driver monitoring camera. The first in-vehicle monitoring camera, the second in-vehicle monitoring camera, and the third in-vehicle monitoring camera are IR-RGB cameras, and the driver monitoring camera is an IR camera. The vehicle cabin monitoring controller receives images collected by the driver's visual monitoring device to monitor the driver and passengers. Based on the image data from the driver's visual monitoring device, the vehicle cabin monitoring controller determines the current ambient brightness. It then compares the current ambient brightness with a set dim light threshold and, based on the result, notifies the driver's visual monitoring device to activate the infrared auxiliary light. The vehicle cabin monitoring controller also sends the current ambient brightness information to the in-vehicle display screen, which adjusts its brightness accordingly. Images are captured by the camera under different light intensities. The RAW data of the images is statistically analyzed to calculate the average values ​​of the G and IR components of the image. The average values ​​of the G and IR components are used to represent the gray-scale mean of the G and IR components of the image, respectively. The mapping relationship between the gray-scale mean of the image data and the light intensity is obtained. Different brightness levels are obtained based on the mapping relationship. The vehicle cabin monitoring controller sends the brightness level of the captured image to the vehicle display screen. The brightness of the vehicle display screen is adaptively adjusted according to the brightness level. The current ambient brightness is compared with the set dimness threshold as follows: The vehicle cabin monitoring controller receives RAW data collected by the camera of the driver's vision monitoring device, performs statistical analysis on each frame of RAW data, and calculates the average value of the G component and IR component of the image. The average value of the G component and IR component is used to represent the grayscale mean of the G component and IR component of the image, respectively. When the average value of G is less than the average value of IR, the light is considered dim, and the infrared supplementary light is turned on. When the average G value is greater than the average IR value, the difference between the G and IR components is calculated and combined with the camera's gain and shutter parameters to obtain a brightness estimate. When the brightness estimate is less than the dim light threshold, the light is considered dim and the infrared fill light is turned on; when the brightness estimate is greater than the dim light threshold, the light is considered normal and the infrared fill light is turned off. The first in-vehicle monitoring camera, the second in-vehicle monitoring camera, the third in-vehicle monitoring camera, and the driver monitoring camera of the driver visual monitoring device share an ISP chip integrated on the SOC chip of the vehicle cockpit monitoring controller for image processing.

2. The adaptive dimming system for automotive cabins based on visual monitoring according to claim 1, characterized in that: The driver monitoring camera is installed on the A-pillar on the driver's side inside the vehicle; the first in-vehicle monitoring camera is installed on the rearview mirror inside the vehicle. The second in-vehicle monitoring camera is installed on the driver's side B-pillar inside the vehicle; the third in-vehicle monitoring camera is installed on the passenger side B-pillar inside the vehicle.

3. The adaptive dimming system for a car cabin based on visual monitoring according to claim 1, characterized in that: The in-vehicle display screens include an instrument panel screen, a central control screen, and a headrest screen.

4. The adaptive dimming system for automotive cabins based on visual monitoring according to claim 1, characterized in that: The first, second, and third in-vehicle monitoring cameras are used to detect the occupant area. The vehicle cabin monitoring controller performs image recognition on the occupant's behavior and actions in the image data collected by the first, second, and third in-vehicle monitoring cameras, and links vehicle control functions based on the recognition results. The vehicle cabin monitoring controller also performs image recognition on the driver in the image data collected by the driver monitoring camera, including facial recognition, fatigue monitoring, and attention monitoring.

5. A method for adaptive dimming in a car cabin based on visual monitoring, characterized in that, The adaptive dimming system for automotive cabins based on visual monitoring, as described in claim 1, includes the following steps: The driver's visual monitoring device collects image data and sends it to the vehicle's cockpit monitoring controller; The vehicle cabin monitoring controller obtains the current ambient brightness based on image data from the driver's visual monitoring device; The vehicle cabin monitoring controller judges the current ambient brightness against the set dim light threshold, and notifies the driver's visual monitoring device to turn on the infrared auxiliary light based on the judgment result. The vehicle cabin monitoring controller sends the current ambient brightness information to the in-vehicle display screen; The in-vehicle display adjusts its brightness based on the received ambient light level.

6. The adaptive dimming method for a car cabin based on visual monitoring according to claim 5, characterized in that: When the driver monitoring camera activates its infrared fill light for infrared exposure, the first, second, and third in-vehicle monitoring cameras are not allowed to perform exposure.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a program for executing the visual monitoring-based adaptive dimming method for automotive cabins as described in claim 5.

Images