Binocular camera module control method, related equipment and vehicle

Abstract

The invention provides a binocular camera module control method, related equipment and a vehicle. The method comprises the following steps: receiving a preset signal sent by a vehicle-mounted host HU; the preset signal comprises a synchronous acquisition trigger signal or a diagnosis signal; in response to the received synchronous acquisition trigger signal, sending the synchronous acquisition trigger signal to the RGB camera and the IR camera; receiving RGB image data and IR image data sent by the RGB camera and the IR camera respectively; processing the RGB image data and the IR image data according to a preset image processing rule; transmitting the processed RGB image data and IR image data to the SOC of the HU; in response to the received diagnosis signal, sending the diagnosis signal to the RGB camera and the IR camera; receiving corresponding fault information fed back by the RGB camera and the IR camera; and fault information is sent to the SOC of the HU.

Description

Technical Field

[0001] The present disclosure relates to the technical field of binocular camera module control, and particularly to a binocular camera module control method, related devices and vehicles. Background Art

[0002] In the related art, inside an intelligent vehicle, a binocular camera module (including a Red Green Blue (RGB) camera and an Infrared (IR) camera) is usually used to collect RGB image data and IR image data inside the vehicle. However, when directly using the RGB camera and the IR camera in the binocular camera module to collect the RGB image data and the IR image data inside the vehicle, it is impossible to ensure the synchronization between the RGB camera and the IR camera when collecting the RGB image data and the IR image data, and there may be a problem that the collected RGB image data or IR image data is incorrect due to a failure of the RGB camera or the IR camera in the binocular camera module. Therefore, it is also necessary to perform fault diagnosis on the RGB camera and the IR camera in the binocular camera module. Summary of the Invention

[0003] The present disclosure provides a binocular camera module control method, related devices and vehicles.

[0004] According to a first aspect of the present disclosure, there is provided a binocular camera module control method, including:

[0005] Receiving a preset signal sent by an in-vehicle host HU; the preset signal includes a synchronous acquisition trigger signal or a diagnostic signal; the synchronous acquisition trigger signal is used to make the Red Green Blue (RGB) camera and the Infrared (IR) camera in the binocular camera module synchronously collect image data inside the vehicle; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly;

[0006] In response to receiving the synchronous acquisition trigger signal, sending the synchronous acquisition trigger signal to the RGB camera and the IR camera;

[0007] Receiving the RGB image data and the IR image data respectively sent by the RGB camera and the IR camera;

[0008] Processing the RGB image data and the IR image data according to a preset image processing rule;

[0009] Sending the processed RGB image data and IR image data to the System-on-Chip (SOC) of the HU;

[0010] In response to receiving the diagnostic signal, sending the diagnostic signal to the RGB camera and the IR camera;

[0011] Receiving the corresponding fault information fed back by the RGB camera and the IR camera;

[0012] And send a fault message to the SOC of the HU.

[0013] In some embodiments of the present disclosure,

[0014] Processing RGB image data according to a preset image processing rule, including:

[0015] Performing a first image preprocessing on the RGB image data in RAW format and outputting the RGB image data in YUV format; the first image preprocessing includes at least one of gamma correction, white balance, color correction, contrast adjustment, saturation adjustment, or region of interest extraction;

[0016] Processing IR image data according to a preset image processing rule, including:

[0017] Performing a second image preprocessing on the IR image data in RAW format and outputting the IR image data in YUV format; the second image preprocessing includes at least one of gamma correction, sharpening adjustment, brightness uniformity adjustment, or region of interest extraction.

[0018] In some embodiments of the present disclosure, sending a synchronous acquisition trigger signal to the RGB camera and the IR camera, including:

[0019] Copying the synchronous acquisition trigger signal;

[0020] Sending the synchronous acquisition trigger signal and the copied synchronous acquisition trigger signal to the RGB camera and the IR camera respectively.

[0021] In some embodiments of the present disclosure, the number of binocular camera modules is one or at least two;

[0022] Sending a synchronous acquisition trigger signal to the RGB camera and the IR camera, including:

[0023] Sending a synchronous acquisition trigger signal to the RGB camera and the IR camera in each binocular camera module.

[0024] According to the second aspect of the present disclosure, there is provided a method for controlling a binocular camera module, including:

[0025] Sending a preset signal to an image signal processing ISP device, the preset signal including a synchronous acquisition trigger signal sent by a micro control unit MCU in an in-vehicle host HU or a diagnostic signal sent by a system-on-chip SOC in the HU; the synchronous acquisition trigger signal is used to make the RGB camera and the IR camera in the binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly;

[0026] In response to sending a synchronous acquisition trigger signal, the processed RGB image data and IR image data sent by the ISP device are received through the SOC;

[0027] In response to sending a diagnostic signal, the fault information sent by the ISP device is received through the SOC; the fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty;

[0028] Based on the fault information, determine whether the corresponding RGB camera and IR camera are faulty.

[0029] In some embodiments of the present disclosure, determining whether the corresponding RGB camera and IR camera are faulty based on the fault information includes:

[0030] Analyze the fault information to obtain a first return value in the fault information for indicating whether the RGB camera is faulty, and obtain a second return value in the fault information for indicating whether the IR camera is faulty; the first return value refers to the return value of the first transmission port corresponding to the RGB camera in the ISP device, and the second return value refers to the return value of the second transmission port corresponding to the IR camera in the ISP device;

[0031] If the first return value is low level, it is determined that the RGB camera is faulty;

[0032] If the second return value is low level, it is determined that the IR camera is faulty.

[0033] In some embodiments of the present disclosure, after analyzing the fault information, the method provided in the second aspect of the present disclosure further includes:

[0034] Obtain a third return value in the fault information for indicating whether the light supplement driving device corresponding to the IR camera is faulty; the third return value refers to the return value of the third transmission port of the light supplement driving device corresponding to the IR camera in the ISP device;

[0035] If both the second return value and the third return value are low level, it is determined that the light supplement driving device corresponding to the IR camera is faulty; the light supplement driving device is used to supplement light for the IR camera.

[0036] In some embodiments of the present disclosure, the binocular camera module control method further includes:

[0037] In response to the vehicle power-on signal, wake-up signal or working signal, power on the ISP device through the MCU;

[0038] In response to the vehicle sleep signal, power off the ISP device through the MCU;

[0039] According to the third aspect of the present disclosure, an Image Signal Processing (ISP) device is provided, including:

[0040] A first receiving unit, configured to receive a preset signal sent by an in-vehicle host HU; the preset signal includes a synchronous acquisition trigger signal or a diagnostic signal; the synchronous acquisition trigger signal is used to make an RGB camera and an IR camera in a binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly;

[0041] A first transmitting unit, configured to send a synchronous acquisition trigger signal to the RGB camera and the IR camera, or send the diagnostic signal to the RGB camera and the IR camera;

[0042] A second receiving unit, configured to receive RGB image data and IR image data respectively sent by the RGB camera and the IR camera, or receive corresponding fault information fed back by the RGB camera and the IR camera, and store the fault information in corresponding addresses of a register; the fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty;

[0043] A processing unit, configured to process the RGB image data and the IR image data according to a preset image processing rule;

[0044] A second transmitting unit, configured to send the processed RGB image data and IR image data to the SOC of the HU, or send the fault information to the SOC of the HU.

[0045] According to a fourth aspect of the present disclosure, there is provided an in-vehicle host HU, including:

[0046] A third transmitting unit, configured to send a preset signal to an image signal processing ISP device, the preset signal includes a synchronous acquisition trigger signal sent through a micro control unit MCU in the in-vehicle host HU or a diagnostic signal sent through a system-on-chip SOC in the HU; the synchronous acquisition trigger signal is used to make an RGB camera and an IR camera in a binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly;

[0047] A third receiving unit, configured to receive, through the SOC, the processed RGB image data and IR image data sent by the ISP device, or receive the fault information sent by the ISP device through the SOC; the fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty;

[0048] A judging unit, configured to judge whether the corresponding RGB camera and IR camera are faulty according to the fault information.

[0049] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0050] At least one processor; and

[0051] A memory communicatively connected to at least one processor; wherein,

[0052] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the foregoing first aspect or the second aspect.

[0053] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method of the foregoing first aspect or the second aspect.

[0054] According to a seventh aspect of the present disclosure, there is provided a vehicle including at least one of the image signal processing ISP device of the foregoing third aspect, the in-vehicle host HU of the fourth aspect, the electronic device of the fifth aspect, or the storage medium of the sixth aspect.

[0055] The present disclosure provides a binocular camera module control method, related devices, and a vehicle. The binocular camera module control method includes: a head unit (HU) sends a preset signal to an image signal process (ISP) device, where the preset signal includes a synchronous acquisition trigger signal sent through a micro control unit (MCU) in the HU, or a diagnostic signal sent through a system on chip (SOC) in the HU; the synchronous acquisition trigger signal is used to enable an RGB camera and an IR camera in the binocular camera module to synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly; the ISP device receives the preset signal sent by the HU; in response to receiving the synchronous acquisition trigger signal, the ISP device sends the synchronous acquisition trigger signal to the RGB camera and the IR camera; the ISP device receives RGB image data and IR image data respectively sent by the RGB camera and the IR camera; the ISP device processes the RGB image data and the IR image data according to preset image processing rules; the ISP device sends the processed RGB image data and IR image data to the SOC of the HU; the HU receives the processed RGB image data and IR image data sent by the ISP device through the SOC; or, in response to receiving the diagnostic signal, the ISP device sends the diagnostic signal to the RGB camera and the IR camera; the ISP device receives corresponding fault information fed back by the RGB camera and the IR camera, and the fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty; the ISP device sends the fault information to the SOC of the HU; the HU receives the fault information sent by the ISP device through the SOC; the SOC of the HU determines whether the corresponding RGB camera and IR camera are faulty according to the fault information.

[0056] According to the solution of the present disclosure, the ISP device receives a synchronous acquisition trigger signal sent by the MCU in the HU or a diagnostic signal sent by the SOC in the HU; in response to receiving the synchronous acquisition trigger signal, the ISP device sends a synchronous acquisition trigger signal or a diagnostic signal to the RGB camera and the IR camera; the ISP device receives the RGB image data and the IR image data respectively sent by the RGB camera and the IR camera, or the fault information fed back by the RGB camera and the IR camera; the ISP device processes the RGB image data and the IR image data according to a preset image processing rule, and sends the processed RGB image data and IR image data to the SOC of the HU, or sends the fault information to the SOC of the HU. In this solution, the ISP device sends a synchronous acquisition trigger signal to the RGB camera and the IR camera in the binocular camera module, which can achieve the purpose of controlling the RGB camera and the IR camera in the binocular camera module to synchronously acquire the internal image data of the vehicle. Or the ISP device sends a diagnostic signal to the RGB camera and the IR camera in the binocular camera module, which can achieve the purpose of judging whether the RGB camera and the IR camera in the binocular camera module are faulty, thereby ensuring the synchronization between the RGB camera and the IR camera when acquiring the RGB image data and the IR image data, and solving the problem that the acquired RGB image data or IR image data is incorrect due to the fault of the RGB camera or the IR camera in the binocular camera module.

[0057] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. Brief Description of the Drawings

[0058] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0059] Figure 1 It is a schematic flowchart of the first binocular camera module control method provided by the embodiment of the present disclosure;

[0060] Figure 2 It is a schematic flowchart of the second binocular camera module control method provided by the embodiment of the present disclosure;

[0061] Figure 3 It is a schematic structural diagram of an ISP device provided by the embodiment of the present disclosure;

[0062] Figure 4 It is a schematic structural diagram of a HU provided by the embodiment of the present disclosure;

[0063] Figure 5 It is a schematic block diagram of an exemplary electronic device provided by the embodiment of the present disclosure. Detailed Description of the Embodiments

[0064] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.

[0065] The following describes a binocular camera module control method, related devices, and vehicles according to embodiments of the present disclosure with reference to the accompanying drawings.

[0066] In the present disclosure, an ISP device receives a synchronous acquisition trigger signal sent by an MCU in an HU or a diagnostic signal sent by an SOC in the HU; in response to receiving the synchronous acquisition trigger signal, the ISP device sends a synchronous acquisition trigger signal or a diagnostic signal to an RGB camera and an IR camera; the ISP device receives RGB image data and IR image data respectively sent by the RGB camera and the IR camera, or fault information fed back by the RGB camera and the IR camera; the ISP device processes the RGB image data and the IR image data according to a preset image processing rule, and sends the processed RGB image data and IR image data to the SOC of the HU, or sends fault information to the SOC of the HU. In this solution, the ISP device sends a synchronous acquisition trigger signal to the RGB camera and the IR camera in the binocular camera module, which can achieve the purpose of controlling the RGB camera and the IR camera in the binocular camera module to synchronously acquire vehicle interior image data, or the ISP device sends a diagnostic signal to the RGB camera and the IR camera in the binocular camera module, which can achieve the purpose of determining whether the RGB camera and the IR camera in the binocular camera module are faulty, thereby ensuring the synchronization between the RGB camera and the IR camera when acquiring RGB image data and IR image data, and solving the problem that the acquired RGB image data or IR image data is incorrect due to a fault in the RGB camera or the IR camera in the binocular camera module.

[0067] The binocular camera module control method provided by the embodiments of the present disclosure can be applied to the following scenarios.

[0068] The present disclosure includes at least one binocular camera module. The binocular camera module includes an RGB camera and an IR camera, and may or may not include an ISP device. Both the RGB camera and the IR camera are communicatively connected to the ISP device; the output end of the ISP device is communicatively connected to the input end of the first data transmission unit, and the number of the first data transmission units is the same as the number of the binocular camera modules; the signal output end of the first data transmission unit is connected to the signal input end of the second data transmission unit; the second data transmission unit is configured to send the image data collected by the RGB camera and the IR camera processed by the ISP module to the SOC of the HU.

[0069] In one embodiment, the Mobile Industry Processor Interface (MIPI) protocol is adopted between the ISP device and the RGB camera, between the ISP device and the IR camera, between the ISP device and the first data transmission unit, and between the second data transmission unit and the SOC of the HU.

[0070] In one embodiment, the second-generation version of the Gigabit Multimedia Serial Link (GMSL2) protocol is adopted for data transmission between the first data transmission unit and the second data transmission unit.

[0071] In one embodiment, the binocular camera module further includes a fill light driving device corresponding to the IR camera, and the fill light driving device is configured to provide fill light for the IR camera.

[0072] In one embodiment, the HU is the core of the human-machine interaction unit of the vehicle, generally referring to the in-vehicle infotainment system and the central control screen, and the cockpit domain controller.

[0073] In one embodiment, the HU includes an MCU and an SOC.

[0074] In one embodiment, the MCU is configured to provide a synchronous acquisition trigger signal for the binocular camera module, the RGB camera, and the IR camera.

[0075] In one embodiment, the synchronous signal output end of the MCU is connected to the synchronous signal input end of the ISP device through the second data transmission unit and the corresponding first data transmission unit.

[0076] In one embodiment, the SOC is configured to provide a diagnostic signal for the RGB camera and the IR camera in the binocular camera module, and the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly.

[0077] In one embodiment, the signal input end of the SOC is connected to the second data transmission unit, the corresponding first data transmission unit, and the diagnostic signal output end of the ISP device.

[0078] In one embodiment, the ISP device can receive the diagnostic signal sent by the HU through the IIC protocol.

[0079] The first binocular camera module control method applied to the ISP device provided by the embodiments of the present disclosure has the ISP device as its execution subject. Specifically, it can be a separate ISP device or an ISP device integrated in the binocular camera module. More specifically, its execution subject can be an ISP chip or a processor integrated with image signal processing functions.

[0080] Figure 1 It is a schematic flowchart of the first binocular camera module control method provided by the embodiments of the present disclosure.

[0081] As Figure 1 shown, the first binocular camera module control method provided by the embodiments of the present disclosure includes the following steps:

[0082] Step 101, receive a preset signal sent by the in-vehicle host HU; the preset signal includes a synchronous acquisition trigger signal or a diagnostic signal; the synchronous acquisition trigger signal is used to make the red, green, and blue (RGB) camera and the infrared (IR) camera in the binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly;

[0083] In one embodiment, the ISP device receives the synchronous acquisition trigger signal sent by the MCU in the HU through the second data transmission unit and the corresponding first data transmission unit.

[0084] In one embodiment, the ISP device receives the diagnostic signal sent by the SOC in the HU through the second data transmission unit and the corresponding first data transmission unit.

[0085] Step 1021, in response to receiving the synchronous acquisition trigger signal, send the synchronous acquisition trigger signal to the RGB camera and the IR camera;

[0086] In one embodiment, the MCU in the HU provides the synchronous acquisition trigger signal for the RGB camera and the IR camera in the corresponding binocular camera module through the second data transmission unit and the corresponding first data transmission unit.

[0087] In one embodiment, the ISP device receives a synchronous acquisition trigger signal sent by the MCU in the HU, duplicates a path of synchronous acquisition trigger signal inside the ISP device, and the ISP device sends the received synchronous acquisition trigger signal and the duplicated synchronous acquisition trigger signal to the RGB camera and the IR camera respectively, so as to control the RGB camera and the IR camera to synchronously acquire the internal RGB image data and IR image data of the vehicle.

[0088] Step 1031, receive the RGB image data and the IR image data sent by the RGB camera and the IR camera respectively;

[0089] In one embodiment, the ISP device receives the RAW format RGB image data and IR image data sent by the RGB camera and the IR camera respectively through the MIPI protocol.

[0090] Step 1041, process the RGB image data and the IR image data according to the preset image processing rules;

[0091] In one embodiment, the ISP device processes the RAW format RGB image data and IR image data according to the preset image processing rules to generate the YUV format RGB image data and IR image data.

[0092] Step 1051, send the processed RGB image data and IR image data to the SOC of the HU;

[0093] In one embodiment, the ISP device sends the processed YUV format RGB image data and IR image data to the SOC of the HU.

[0094] Step 1022, in response to receiving the diagnostic signal, send the diagnostic signal to the RGB camera and the IR camera;

[0095] In one embodiment, the diagnostic signal sent by the SOC in the HU is transmitted to the ISP device after passing through the second data transmission unit and the first data transmission unit.

[0096] In one embodiment, after the ISP device receives the diagnostic signal sent by the SOC in the HU, the ISP device sends the diagnostic signal to the RGB camera and the IR camera.

[0097] Step 1032, receive the corresponding fault information fed back by the RGB camera and the IR camera;

[0098] In one embodiment, the ISP device receives the corresponding fault information fed back by the RGB camera and the IR camera.

[0099] In one embodiment, the fault information of the RGB camera or the IR camera includes at least one of the fault information such as sensor freeze frame, abnormal power supply, thermostat error reporting, abnormal power chip, and MIPI transmission anomaly.

[0100] Step 1042, and send the fault information to the SOC of the HU.

[0101] In one embodiment, the ISP device transmits the RGB camera and IR camera fault information to the SOC in the HU through the first data transmission unit and the second data transmission unit.

[0102] In one embodiment, the ISP device decides whether to execute steps 1021 to 1051 or steps 1022 to 1042 subsequently according to whether the received preset signal is a synchronous acquisition trigger signal or a diagnostic signal.

[0103] In one embodiment, step 1041 includes:

[0104] Process the RGB image data according to the preset image processing rules, including:

[0105] Perform first image preprocessing on the RAW format RGB image data, and output the RGB image data in YUV format.

[0106] In one embodiment, the first image preprocessing includes at least one of gamma correction, white balance, color correction, contrast adjustment, saturation adjustment, or region of interest extraction;

[0107] In one embodiment, the ISP device performs first image preprocessing on the RAW format RGB image data collected by the RGB camera.

[0108] In one embodiment, in the YUV format, the Y channel represents the luminance signal in the image data, and the U channel and the V channel represent the chrominance signals in the image data.

[0109] Process the IR image data according to the preset image processing rules, including:

[0110] Perform second image preprocessing on the RAW format IR image data, and output the IR image data in YUV format;

[0111] In one embodiment, the second image preprocessing includes at least one of gamma correction, sharpening adjustment, brightness uniformity adjustment, or region of interest extraction.

[0112] In one embodiment, the ISP device performs second image preprocessing on the RAW format IR image data collected by the IR camera.

[0113] In one embodiment, step 1021 includes:

[0114] Copy the synchronous acquisition trigger signal;

[0115] In one embodiment, after the ISP device receives the synchronous acquisition trigger signal sent by the MCU in the HU, the ISP device copies a path of synchronous acquisition trigger signal inside.

[0116] In one embodiment, each ISP device copies a path of synchronous acquisition trigger signal inside.

[0117] Send the synchronous acquisition trigger signal and the copied synchronous acquisition trigger signal to the RGB camera and the IR camera respectively.

[0118] In one embodiment, send the synchronous acquisition trigger signal and the copied synchronous acquisition trigger signal in each ISP device to the RGB camera and the IR camera corresponding to each ISP device respectively.

[0119] In one embodiment, the number of binocular camera modules is one or at least two;

[0120] Sending the synchronous acquisition trigger signal to the RGB camera and the IR camera includes:

[0121] Send the synchronous acquisition trigger signal to the RGB camera and the IR camera in each binocular camera module.

[0122] In one embodiment, if the number of binocular camera modules is one, the binocular camera module can be arranged at the corresponding position according to the acquisition requirement. If the number of binocular camera modules is two, the two binocular camera modules are respectively arranged at the corresponding positions of the front row cockpit and the rear row cockpit of the vehicle; if the number of binocular camera modules is multiple, the multiple binocular camera modules are arranged at different positions inside the vehicle to collect richer image data.

[0123] In one embodiment, the ISP device sends the synchronous acquisition trigger signal to the RGB camera and the IR camera in each binocular camera module, and controls the RGB camera and the IR camera to synchronously acquire the RGB image data and the IR image data inside the vehicle.

[0124] As Figure 2 shown, the second binocular camera module control method provided by the embodiment of the present disclosure, the execution subject of this method is the HU, and the HU includes an MCU and an SOC. This method includes the following steps:

[0125] Step 201: Send a preset signal to the ISP device. The preset signal includes a synchronous acquisition trigger signal sent by the micro control unit (MCU) in the in-vehicle host HU or a diagnostic signal sent by the system on a chip (SOC) in the HU. The synchronous acquisition trigger signal is used to make the red, green, and blue (RGB) camera and the infrared (IR) camera in the binocular camera module synchronously acquire vehicle interior image data. The diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly.

[0126] Step 2021: In response to sending the synchronous acquisition trigger signal, receive the processed RGB image data and IR image data sent by the ISP device through the SOC.

[0127] Step 2022: In response to sending the diagnostic signal, receive the fault information sent by the ISP device through the SOC. The fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty.

[0128] Step 2032: Based on the fault information, determine whether the corresponding RGB camera and IR camera are faulty.

[0129] In one embodiment, if the MCU in the HU sends a synchronous acquisition trigger signal, the SOC receives the processed RGB image data and IR image data sent by the ISP device through the first data transmission unit and the second data transmission unit.

[0130] In one embodiment, if the SOC in the HU sends a diagnostic signal, the SOC receives the fault information sent by the ISP device through the first data transmission unit and the second data transmission unit.

[0131] In one embodiment, based on the corresponding fault information fed back by the RGB camera and the IR camera, determine whether the corresponding RGB camera and IR camera are faulty.

[0132] Specifically, in one embodiment, step 2032 includes:

[0133] Analyze the fault information to obtain a first return value in the fault information for indicating whether the RGB camera is faulty, and obtain a second return value in the fault information for indicating whether the IR camera is faulty. The first return value refers to the return value of the first transmission port corresponding to the RGB camera in the ISP device, and the second return value refers to the return value of the second transmission port corresponding to the IR camera in the ISP device.

[0134] If the first return value is low level, determine that the RGB camera is faulty.

[0135] If the second return value is low level, determine that the IR camera is faulty.

[0136] In one embodiment, after parsing the fault information, the second binocular camera module control method provided by the embodiments of the present disclosure further includes:

[0137] Obtaining a third return value in the fault information for indicating whether the fill light driving device corresponding to the IR camera is faulty; the third return value refers to the return value of the third transmission port of the fill light driving device corresponding to the IR camera in the ISP device;

[0138] If both the second return value and the third return value are low levels, it is determined that the fill light driving device corresponding to the IR camera is faulty; the fill light driving device is used to fill light for the IR camera.

[0139] In one embodiment, by parsing the corresponding fault information fed back by the RGB camera and the IR camera, a first return value for indicating whether the RGB camera is faulty, a second return value for indicating whether the IR camera is faulty, and a third return value are obtained respectively.

[0140] In one embodiment, when faults such as sensor freeze frame, abnormal power supply, temperature warning, abnormal power chip, and MIPI transmission abnormality occur in the RGB camera or the IR camera, the first return value for indicating whether the RGB camera is faulty, the second return value for indicating whether the IR camera is faulty, and the third return value will be pulled low to a low level.

[0141] In one embodiment, according to whether the first return value, the second return value, and the third return value are low levels, it is determined whether the corresponding RGB camera and IR camera are faulty.

[0142] In one embodiment, if the first return value for indicating whether the RGB camera is faulty is low level, it is determined that the RGB camera is faulty; if the second return value for indicating whether the IR camera is faulty is low level, it is determined that the IR camera is faulty; if both the second return value and the third return value for indicating whether the IR camera is faulty are low levels, it is determined that the fill light driving device corresponding to the IR camera is faulty.

[0143] In one embodiment, the second binocular camera module control method further includes:

[0144] In response to the vehicle power-on signal, wake-up signal or working signal, power on the ISP device through the MCU;

[0145] In response to the vehicle sleep signal, power off the ISP device through the MCU;

[0146] In one embodiment, in response to the vehicle power-on signal, control the vehicle's Battery Management System (BMS) to power on the ISP device through the MCU.

[0147] In one embodiment, in response to the vehicle's sleep signal, the MCU controls the vehicle's BMS to power off the ISP device.

[0148] In one embodiment, in response to the vehicle's wake-up signal, the MCU controls the vehicle's BMS to power on the ISP device.

[0149] In one embodiment, in response to the vehicle's working signal, the MCU controls the vehicle's BMS to power on the ISP device.

[0150] The binocular camera module control method provided by the embodiments of the present disclosure includes: the HU sends a preset signal to the ISP device, and the preset signal includes a synchronous acquisition trigger signal sent by the MCU in the HU or a diagnostic signal sent by the SOC in the HU; the synchronous acquisition trigger signal is used to make the RGB camera and the IR camera in the binocular camera module synchronously acquire the internal image data of the vehicle; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly; the ISP device receives the preset signal sent by the HU; in response to receiving the synchronous acquisition trigger signal, the ISP device sends the synchronous acquisition trigger signal to the RGB camera and the IR camera; the ISP device receives the RGB image data and the IR image data sent by the RGB camera and the IR camera respectively; the ISP device processes the RGB image data and the IR image data according to the preset image processing rules; the ISP device sends the processed RGB image data and IR image data to the SOC of the HU; the HU receives the processed RGB image data and IR image data sent by the ISP device through the SOC; or, in response to receiving the diagnostic signal, the ISP device sends the diagnostic signal to the RGB camera and the IR camera; the ISP device receives the corresponding fault information fed back by the RGB camera and the IR camera, and the fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty; the ISP device sends the fault information to the SOC of the HU; the HU receives the fault information sent by the ISP device through the SOC; the SOC of the HU determines whether the corresponding RGB camera and IR camera are faulty according to the fault information.

[0151] According to the solution of the present disclosure:

[0152] First, the ISP device receives a synchronous acquisition trigger signal sent by the MCU in the HU or a diagnostic signal sent by the SOC in the HU; in response to receiving the synchronous acquisition trigger signal, the ISP device sends a synchronous acquisition trigger signal or a diagnostic signal to the RGB camera and the IR camera; the ISP device receives the RGB image data and the IR image data sent by the RGB camera and the IR camera respectively, or the fault information fed back by the RGB camera and the IR camera; the ISP device processes the RGB image data and the IR image data according to the preset image processing rules, and sends the processed RGB image data and IR image data to the SOC of the HU, or sends the fault information to the SOC of the HU. In this solution, the ISP device sends a synchronous acquisition trigger signal to the RGB camera and the IR camera in the binocular camera module, which can achieve the purpose of controlling the RGB camera and the IR camera in the binocular camera module to synchronously acquire the internal image data of the vehicle, or the ISP device sends a diagnostic signal to the RGB camera and the IR camera in the binocular camera module, which can achieve the purpose of judging whether the RGB camera and the IR camera in the binocular camera module are faulty, thereby ensuring the synchronization between the RGB camera and the IR camera when acquiring the RGB image data and the IR image data, and solving the problem that the acquired RGB image data or IR image data is incorrect due to the fault of the RGB camera or the IR camera in the binocular camera module.

[0153] Secondly, by processing the RAW format RGB image data and IR image data according to the preset image processing rules, YUV format RGB image data and IR image data are generated. Since the human eye is much more sensitive to the luminance information (Y) than the chrominance information (U and V), different degrees of compression can be performed on the U and V components without affecting the visual effect. This characteristic enables the ISP device to have higher compression efficiency when storing and transmitting the YUV format RGB image data and IR image data.

[0154] Figure 3 The structural schematic diagram of the ISP device provided by the embodiment of the present disclosure is shown as Figure 3 shown, the ISP device 300 includes:

[0155] A first receiving unit 301, configured to receive a preset signal sent by the in-vehicle host HU; the preset signal includes a synchronous acquisition trigger signal or a diagnostic signal; the synchronous acquisition trigger signal is used to make the RGB camera and the IR camera in the binocular camera module synchronously acquire the internal image data of the vehicle; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly;

[0156] A first sending unit 302, configured to send a synchronous acquisition trigger signal to the RGB camera and the IR camera, or send the diagnostic signal to the RGB camera and the IR camera;

[0157] A second receiving unit 303, configured to receive RGB image data and IR image data respectively sent by an RGB camera and an IR camera, or receive corresponding fault information fed back by the RGB camera and the IR camera, and store the fault information into corresponding addresses of a register; the fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty;

[0158] A processing unit 304, configured to process the RGB image data and the IR image data according to a preset image processing rule;

[0159] A second sending unit 305, configured to send the processed RGB image data and IR image data to the SOC of the HU, or send the fault information to the SOC of the HU.

[0160] In one embodiment, the processing unit 304 is specifically configured to:

[0161] Perform a first image preprocessing on the RGB image data in RAW format, and output RGB image data in YUV format; the first image preprocessing includes at least one of gray coefficient correction, white balance, color correction, contrast adjustment, saturation adjustment, or region of interest extraction;

[0162] Perform a second image preprocessing on the IR image data in RAW format, and output IR image data in YUV format; the second image preprocessing includes at least one of gray coefficient correction, sharpening adjustment, brightness uniformity adjustment, or region of interest extraction.

[0163] In one embodiment, the first sending unit 302 is specifically configured to:

[0164] Copy a synchronous acquisition trigger signal;

[0165] Send the synchronous acquisition trigger signal and the copied synchronous acquisition trigger signal to the RGB camera and the IR camera respectively.

[0166] In one embodiment, the number of binocular camera modules is one or at least two;

[0167] The corresponding ISP device sending a synchronous acquisition trigger signal to the RGB camera and the IR camera includes:

[0168] Sending a synchronous acquisition trigger signal to the RGB camera and the IR camera in each binocular camera module.

[0169] Figure 4 A schematic structural diagram of an HU provided by an embodiment of the present disclosure, as Figure 4 shown, the HU 400 includes:

[0170] A third sending unit 401 is configured to send a preset signal to an Image Signal Processing (ISP) device. The preset signal includes a synchronous acquisition trigger signal sent by a Micro Control Unit (MCU) in a Head Unit (HU) of a vehicle or a diagnostic signal sent by a System on Chip (SOC) in the HU. The synchronous acquisition trigger signal is used to enable an RGB camera and an IR camera in a binocular camera module to synchronously acquire internal vehicle image data. The diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly.

[0171] A third receiving unit 402 is configured to receive, through the SOC, the processed RGB image data and IR image data sent by the ISP device, or receive, through the SOC, fault information sent by the ISP device. The fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty.

[0172] A judging unit 403 is configured to judge, according to the fault information, whether the corresponding RGB camera and IR camera are faulty.

[0173] In an embodiment, the judging unit 403 is specifically configured to:

[0174] Analyze the fault information to obtain a first return value in the fault information for indicating whether the RGB camera is faulty, and obtain a second return value and a third return value in the fault information for indicating whether the IR camera is faulty. The first return value refers to the return value of the transmission port corresponding to the RGB camera in the ISP device. The second return value and the third return value refer to the return values of the second port and the third port of the ISP device corresponding to the IR camera and the light supplement driving device corresponding to the IR camera.

[0175] If the first return value is a low level, it is determined that the RGB camera is faulty.

[0176] If the second return value is a low level, it is determined that the IR camera is faulty.

[0177] If both the second return value and the third return value are low levels, it is determined that the light supplement driving device corresponding to the IR camera is faulty. The light supplement driving device is used to supplement light for the IR camera.

[0178] In an embodiment, the HU 400 further includes a control unit, and the control unit is configured to:

[0179] Power on the ISP device through the MCU in response to a vehicle power-on signal, a wake-up signal or a working signal.

[0180] Power off the ISP device through the MCU in response to a vehicle sleep signal.

[0181] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a non-transitory computer-readable storage medium storing computer instructions.

[0182] Specifically, an embodiment of the present disclosure provides an electronic device, including:

[0183] At least one processor; and

[0184] A memory communicatively connected to the at least one processor; wherein,

[0185] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of the binocular camera module control method as described above.

[0186] An embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the steps of the binocular camera module control method as described above.

[0187] An embodiment of the present disclosure provides a vehicle, wherein the vehicle includes at least one of the foregoing ISP device, the foregoing HU, the foregoing electronic device, or the foregoing storage medium.

[0188] Figure 5 FIG. shows a schematic block diagram of an exemplary electronic device 500 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, a vehicle-mounted device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] As Figure 5 shown, the electronic device 500 includes a computing unit 501, which can execute various appropriate actions and processes according to the computer program stored in a ROM (Read-Only Memory) 502 or the computer program loaded from a storage unit 508 into a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0190] Multiple components in device 500 are connected to I / O interface 505, including: input unit 504, such as a keyboard, mouse, etc.; output unit 507, such as various types of displays, speakers, etc.; storage unit 508, such as a disk, optical disc, etc.; and communication unit 509, such as a network card, modem, wireless communication transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0191] Computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit 501 include, but are not limited to, CPU (Central Processing Unit), GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any suitable processor, controller, microcontroller, etc. Computing unit 501 executes the various methods and processes described above, such as the binocular camera module control method. For example, in some embodiments, the binocular camera module control method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by computing unit 501, one or more steps of the method described above can be performed. Alternatively, in other embodiments, computing unit 501 can be configured to execute the aforementioned binocular camera module control method in any other suitable way (e.g., by means of firmware).

[0192] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chip), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0193] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0194] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory), or a flash memory, an optical fiber, a CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and no limitations are imposed herein.

[0196] The above specific embodiments do not constitute a limitation on the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A method for controlling a binocular camera module, characterized in that including: receiving a preset signal sent by an in-vehicle host HU; the preset signal includes a synchronous acquisition trigger signal or a diagnostic signal; the synchronous acquisition trigger signal is used to make the red, green, and blue (RGB) camera and the infrared (IR) camera in the binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly; in response to receiving the synchronous acquisition trigger signal, sending the synchronous acquisition trigger signal to the RGB camera and the IR camera; receiving RGB image data and IR image data sent by the RGB camera and the IR camera respectively; processing the RGB image data and the IR image data according to a preset image processing rule; sending the processed RGB image data and IR image data to the system-on-chip (SOC) of the HU; in response to receiving the diagnostic signal, sending the diagnostic signal to the RGB camera and the IR camera; receiving corresponding fault information fed back by the RGB camera and the IR camera; and sending the fault information to the SOC of the HU.

2. The method according to claim 1, wherein processing the RGB image data according to the preset image processing rule includes: performing a first image preprocessing on the RGB image data in RAW format and outputting the RGB image data in YUV format; the first image preprocessing includes at least one of gamma correction, white balance, color correction, contrast adjustment, saturation adjustment, or region of interest extraction; processing the IR image data according to the preset image processing rule includes: performing a second image preprocessing on the IR image data in RAW format and outputting the IR image data in YUV format; the second image preprocessing includes at least one of gamma correction, sharpening adjustment, brightness uniformity adjustment, or region of interest extraction.

3. The method according to claim 1, wherein Sending the synchronous acquisition trigger signal to the RGB camera and the IR camera includes: duplicating the synchronous acquisition trigger signal; sending the synchronous acquisition trigger signal and the duplicated synchronous acquisition trigger signal to the RGB camera and the IR camera respectively.

4. The method according to claim 1, wherein The number of the binocular camera modules is one or at least two; Sending the synchronous acquisition trigger signal to the RGB camera and the IR camera includes: sending the synchronous acquisition trigger signal to the RGB camera and the IR camera in each binocular camera module.

5. A binocular camera module control method, characterized in that, including: sending a preset signal to an image signal processing (ISP) device, the preset signal includes a synchronous acquisition trigger signal sent by a micro control unit (MCU) in the in-vehicle host HU or a diagnostic signal sent by the system-on-chip (SOC) in the HU; the synchronous acquisition trigger signal is used to make the red, green, and blue (RGB) camera and the infrared (IR) camera in the binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly; In response to sending a synchronous acquisition trigger signal, the processed RGB image data and the IR image data sent by the ISP device are received through the SOC; In response to sending a diagnostic signal, the fault information sent by the ISP device is received through the SOC; The fault information is used to indicate whether the corresponding RGB camera and the IR camera are faulty; Based on the fault information, it is determined whether the corresponding RGB camera and the IR camera are faulty.

6. The method according to claim 5, characterized in that The determining whether the corresponding RGB camera and the IR camera are faulty according to the fault information includes: Analyze the fault information to obtain a first return value in the fault information for indicating whether the RGB camera is faulty, and obtain a second return value in the fault information for indicating whether the IR camera is faulty; the first return value refers to the return value of the first transmission port corresponding to the RGB camera in the ISP device, and the second return value refers to the return value of the second transmission port corresponding to the IR camera in the ISP device; If the first return value is low level, it is determined that the RGB camera is faulty; If the second return value is low level, it is determined that the IR camera is faulty.

7. The method according to claim 6, wherein After analyzing the fault information, the method further includes: Obtain a third return value in the fault information for indicating whether the fill light driving device corresponding to the IR camera is faulty; the third return value refers to the return value of the third transmission port of the fill light driving device corresponding to the IR camera in the ISP device; If both the second return value and the third return value are low level, it is determined that the fill light driving device corresponding to the IR camera is faulty; the fill light driving device is used to fill light for the IR camera.

8. The method according to claim 5, wherein The method further includes: In response to the vehicle power-on signal, wake-up signal or working signal, power on the ISP device through the MCU; In response to the vehicle sleep signal, power off the ISP device through the MCU.

9. An image signal processing ISP device, characterized in that, It includes: A first receiving unit, configured to receive a preset signal sent by the in-vehicle host HU; The preset signal includes a synchronous acquisition trigger signal or a diagnostic signal; The synchronous acquisition trigger signal is used to make the red, green, and blue RGB cameras and the infrared IR camera in the binocular camera module synchronously acquire vehicle interior image data; The diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly; A first sending unit, configured to send the synchronous acquisition trigger signal to the RGB camera and the IR camera, or send the diagnostic signal to the RGB camera and the IR camera; A second receiving unit, configured to receive the RGB image data and the IR image data sent by the RGB camera and the IR camera respectively, or receive the corresponding fault information fed back by the RGB camera and the IR camera, and store the fault information in the corresponding address of the register; The fault information is used to indicate whether the corresponding RGB camera and the IR camera are faulty; A processing unit for processing the RGB image data and the IR image data according to preset image processing rules; A second sending unit for sending the processed RGB image data and the IR image data to the system-on-chip (SOC) of the HU, or sending the fault information to the SOC of the HU.

10. A vehicle-mounted host HU, characterized in that, Comprising: A third sending unit for sending a preset signal to an image signal processing (ISP) device, where the preset signal includes a synchronous acquisition trigger signal sent by a micro control unit (MCU) in an in-vehicle host (HU) or a diagnostic signal sent by the SOC in the HU; the synchronous acquisition trigger signal is used to make the RGB camera and the IR camera in the binocular camera module synchronously acquire vehicle interior image data; the diagnostic signal is used to diagnose whether the RGB camera and the IR camera are working properly; A third receiving unit for receiving, through the SOC, the processed RGB image data and the IR image data sent by the ISP device, or receiving, through the SOC, the fault information sent by the ISP device; The fault information is used to indicate whether the corresponding RGB camera and IR camera are faulty; A judging unit for judging whether the corresponding RGB camera and IR camera are faulty according to the fault information.

11. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 4, or claims 5 to 8.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4, or claims 5 to 8.

13. A vehicle, characterized in that, Comprising at least one of the image signal processing (ISP) device according to claim 9, the in-vehicle host (HU) according to claim 10, the electronic device according to claim 11, or the storage medium according to claim 12.

Images