Data processing system and vehicle

Abstract

The invention provides a data processing system and a vehicle. The system comprises: at least one binocular camera module, wherein the binocular camera module comprises an RGB camera, an IR camera and an ISP module; a first data transmission unit; the second data transmission unit is used for sending the image data processed by the ISP module to the SOC; and the MCU is used for providing a synchronous acquisition trigger signal for the binocular camera module. According to the scheme, the binocular camera module is adopted, and compared with a traditional monocular camera, the RGB component and the IR component occupy all resolutions of the RGB camera and the IR camera respectively, so that the binocular camera module can collect RGB image data and IR image data with higher resolutions; and the RGB image data and the IR image data with higher resolution are processed by the ISP module and then are transmitted to the SOC through the first data transmission unit and the second data transmission unit.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image data processing, and particularly to a data processing system and a vehicle. Background Art

[0002] Currently, the application of intelligent vehicles is becoming more and more widespread. On intelligent vehicles, taking the function of the Driver Monitor System (DMS) as an example, in order to accurately determine whether the driver has behaviors such as fatigue driving and distracted driving based on the driver's behaviors, it is necessary to collect high-clarity image data containing the driver's behaviors. Taking the gesture control function as another example, in order to accurately determine the control requirements of the driver and passengers based on the gestures of the driver and passengers, it is also necessary to collect high-clarity image data containing the gestures of the driver and passengers.

[0003] In the related art, usually, by setting multiple monocular Red Green Blue-Infrared (RGB-IR) cameras inside the vehicle, the monocular RGB-IR cameras are used to collect the image data of the driver and passengers. However, on the sensor of the monocular RGB-IR camera, whether it is the RGB component or the IR component, it only occupies a part of the resolution of the sensor of the RGB-IR camera, resulting in the monocular RGB-IR camera being unable to collect high-clarity image data. Summary of the Invention

[0004] The present disclosure provides a data processing system and a vehicle.

[0005] According to a first aspect of the present disclosure, there is provided a data processing system, including:

[0006] At least one binocular camera module, the binocular camera module includes an RGB camera, an IR camera, and an ISP module; both the RGB camera and the IR camera are communicatively connected to the ISP module; the RGB camera and the IR camera are used to synchronously collect the image data inside the vehicle in response to a synchronous acquisition trigger signal; the ISP module is used to process the image data collected by the RGB camera and the IR camera;

[0007] At least one first data transmission unit having the same number as the binocular camera modules, the signal output end of the ISP module of each binocular camera module is connected to the signal input end of the corresponding first data transmission unit;

[0008] A second data transmission unit, the multiple signal input ends of the second data transmission unit are respectively connected to the signal output ends of each first data transmission unit; the second data transmission unit is used to send the image data collected by the RGB camera and the IR camera processed by the ISP module to the SOC of the in-vehicle host HU;

[0009] HU, where HU includes an MCU and an SOC;

[0010] The synchronization signal output terminal of the MCU is connected to the synchronization signal input terminal of the second data transmission unit; the MCU is used to provide a synchronization acquisition trigger signal for the corresponding binocular camera module through the second data transmission unit and the corresponding first data transmission unit; the synchronization acquisition trigger signal is used to trigger each binocular camera module in at least one binocular camera module to synchronously expose;

[0011] The signal input terminal of the SOC is connected to the signal output terminal of the second data transmission unit.

[0012] In some embodiments of the present disclosure, the binocular camera module further includes an IR fill light driving module and an IR fill light;

[0013] The signal input terminal of the IR fill light driving module is connected to the signal output terminal of the IR camera. The IR fill light driving module is used to send a driving signal to the IR fill light under the control of the turn-on signal output by the IR camera; the driving signal is used to turn on the IR fill light;

[0014] The control terminal of the IR fill light is connected to the signal output terminal of the IR fill light driving module. The IR fill light is used to fill light for the IR camera.

[0015] In some embodiments of the present disclosure, in the light waveform output by the fill light, the total proportion of the rising edge and the falling edge is not greater than 20%, and the proportion difference of the peak waveform does not exceed 10%.

[0016] In some embodiments of the present disclosure, the first data transmission unit includes:

[0017] A serializer group, where the serializer group includes a first serializer and a second serializer;

[0018] A Fakra connector group, where the Fakra connector group includes a first Fakra connector and a second Fakra connector;

[0019] The signal input terminal of the first serializer is connected to the first signal output terminal of the ISP module of the corresponding binocular camera module. The signal output terminal of the first serializer is connected to the signal input terminal of the corresponding first Fakra connector; the first serializer is used to transmit the image data collected by the RGB camera processed by the corresponding ISP module to the corresponding first Fakra connector;

[0020] The signal input end of the second serializer is connected to the second signal output end of the ISP module of the corresponding binocular camera module, and the signal output end of the second serializer is connected to the signal input end of the corresponding second Fakra connector; the second serializer is used to transmit the image data collected by the IR camera processed by the corresponding ISP module to the corresponding second Fakra connector.

[0021] In some embodiments of the present disclosure, the second data transmission unit includes:

[0022] A multi-in-one Fakra connector, and multiple signal input ends of the multi-in-one Fakra connector are respectively connected to the signal output ends of the first Fakra connector and the second Fakra connector of each first data transmission unit;

[0023] A deserializer, the image signal input end of the deserializer is connected to the signal output end of the multi-in-one Fakra connector, and the image signal output end of the deserializer is connected to the signal input end of the SOC.

[0024] In some embodiments of the present disclosure, the SOC includes a diagnostic module;

[0025] The diagnostic module is communicatively connected to the GPIO3 port of the deserializer of the second data transmission unit;

[0026] The diagnostic module is used to send diagnostic instructions to each binocular camera module through the second data transmission unit and the corresponding first data transmission unit, and judge whether each binocular camera module is faulty according to the return value of the GPIO3 port of the deserializer.

[0027] In some embodiments of the present disclosure, between the first serializer and the corresponding first Fakra connector, between the second serializer and the corresponding second Fakra connector, between the first Fakra connector and the second Fakra connector and the multi-in-one Fakra connector, and between the multi-in-one Fakra connector and the deserializer are all connected by coaxial cables;

[0028] Between the ISP module in each binocular camera module and the corresponding first serializer and second serializer, the image data processed by the ISP module is transmitted through the MIPI protocol.

[0029] In some embodiments of the present disclosure, between the first serializer and the corresponding first Fakra connector, between the second serializer and the corresponding second Fakra connector, between the first Fakra connector and the second Fakra connector and the multi-in-one Fakra connector, and between the multi-in-one Fakra connector and the deserializer are all through the GMSL2 protocol to transmit the image data processed by the ISP module at a preset bandwidth transmission rate.

[0030] In some embodiments of the present disclosure, the HU further includes a power management unit;

[0031] The signal input end of the power management unit is connected to the power signal output end of the deserializer, and the power signal output end of the power management unit is connected to the signal input end of the all-in-one Fakra connector; the power management unit is used to power each binocular camera module through the second data transmission unit and the corresponding first data transmission unit.

[0032] In some embodiments of the present disclosure, the RGB camera and IR camera in each binocular camera module respectively send the collected image data in RAW format to the corresponding ISP module. After the ISP module processes the collected image data by the RGB camera and the IR camera respectively, it outputs the image data in YUV format to the corresponding first serializer and second serializer.

[0033] In some embodiments of the present disclosure, the depth of field of the RGB camera and the IR camera in the binocular camera module is 0.2 meters to 3 meters.

[0034] In some embodiments of the present disclosure, the frame rate of the image data collected by the RGB camera and the IR camera in the binocular camera module is not less than 30fps.

[0035] According to a second aspect of the present disclosure, there is provided a vehicle, comprising the data processing system of the first aspect;

[0036] Wherein, if the number of binocular camera modules in the data processing system is two, the two binocular camera modules are respectively arranged at corresponding positions of the front cockpit and the rear cockpit of the vehicle;

[0037] The MCU is used to provide a synchronous acquisition trigger signal for the two binocular camera modules through the second data transmission unit and the corresponding first data transmission unit, so that the two binocular camera modules are exposed synchronously.

[0038] The present disclosure provides a data processing system and a vehicle. The system includes: at least one binocular camera module, where the binocular camera module includes an RGB camera, an IR camera, and an ISP module; both the RGB camera and the IR camera are communicatively connected to the ISP module; the RGB camera and the IR camera are configured to synchronously collect image data inside the vehicle in response to a synchronous acquisition trigger signal; the ISP module is configured to process the image data collected by the RGB camera and the IR camera; at least one first data transmission unit having the same number as the binocular camera modules, and the signal output end of the ISP module of each binocular camera module is connected to the signal input end of the corresponding first data transmission unit; a second data transmission unit, and multiple signal input ends of the second data transmission unit are respectively connected to the signal output ends of each first data transmission unit; the second data transmission unit is configured to send the image data collected by the RGB camera and the IR camera processed by the ISP module to the system-on-chip (SOC) of the in-vehicle host HU; the HU includes an MCU and an SOC; the synchronous signal output end of the MCU is connected to the synchronous signal input end of the second data transmission unit; the MCU is configured to provide a synchronous acquisition trigger signal for the corresponding binocular camera module through the second data transmission unit and the corresponding first data transmission unit; the synchronous acquisition trigger signal is used to trigger each binocular camera module in at least one binocular camera module to synchronously expose; the signal input end of the SOC is connected to the signal output end of the second data transmission unit.

[0039] According to the solution of the present disclosure, the MCU sends a synchronous acquisition trigger signal to each binocular camera module to control each RGB camera and each IR camera in each binocular camera module to synchronously collect image data inside the vehicle; the image data collected by the RGB camera and the IR camera processed by the image signal process (ISP) module in each binocular camera module is sent to the system-on-chip (SOC) of the in-vehicle host (Head Unit, HU) through the first data transmission unit and the second data transmission unit. The camera module in this solution uses a binocular camera module. Compared with the traditional monocular camera, the RGB component and the IR component corresponding to the RGB camera and the IR camera in the binocular camera module respectively occupy the full resolution of the RGB camera and the IR camera, enabling the binocular camera module to collect RGB image data and IR image data with higher resolution; the RGB image data and IR image data with higher resolution, after being processed by the ISP module and then transmitted through the first data transmission unit and the second data transmission unit to the SOC, can present clearer images.

[0040] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. Brief Description of the Drawings

[0041] The drawings are used to better understand the present solution and do not constitute a limitation to the present disclosure. Among them:

[0042] Figure 1 is a schematic structural diagram of the data processing system provided by the embodiments of the present disclosure. Detailed Embodiments

[0043] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist in understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] Embodiments of the present disclosure provide a data processing system and a vehicle.

[0045] As Figure 1 shown, the data processing system provided by the present disclosure includes:

[0046] At least one binocular camera module 1, the binocular camera module 1 includes an RGB camera, an IR camera, and an ISP module; both the RGB camera and the IR camera are communicatively connected to the ISP module; the RGB camera and the IR camera are used to synchronously collect image data inside the vehicle in response to a synchronous acquisition trigger signal; the ISP module is used to process the image data collected by the RGB camera and the IR camera;

[0047] At least one first data transmission unit 2 having the same number as the binocular camera module 1, and the signal output end of the ISP module of each binocular camera module 1 is connected to the signal input end of the corresponding first data transmission unit 2;

[0048] A second data transmission unit 3, and multiple signal input ends of the second data transmission unit 3 are respectively connected to the signal output ends of each first data transmission unit 2; the second data transmission unit 3 is used to send the image data collected by the RGB camera and the IR camera processed by the ISP module to the SOC of the in-vehicle host HU4;

[0049] HU4, HU4 includes an MCU and an SOC;

[0050] The synchronization signal output terminal of the MCU is connected to the synchronization signal input terminal of the second data transmission unit 3; the MCU is configured to provide a synchronization acquisition trigger signal for the corresponding binocular camera module 1 through the second data transmission unit 3 and the corresponding first data transmission unit 2; the synchronization acquisition trigger signal is used to trigger each binocular camera module 1 in at least one binocular camera module 1 to synchronously expose.

[0051] The signal input terminal of the SOC is connected to the signal output terminal of the second data transmission unit 3.

[0052] In one embodiment, the number of the binocular camera modules 1 can be one, two, or more. If the number of the binocular camera modules 1 is two, the two binocular camera modules 1 are respectively arranged at corresponding positions in the front row cockpit and the rear row cockpit of the vehicle; if the number of the binocular camera modules 1 is more, the multiple binocular camera modules 1 are arranged at different positions inside the vehicle to collect richer image data.

[0053] In one embodiment, the RGB camera is used to collect color mode image data inside the vehicle, that is, RGB image data. The IR camera is used to collect near-infrared image data inside the vehicle, that is, IR image data.

[0054] In one embodiment, since the binocular camera module 1 includes an RGB camera and an IR camera, the RGB camera and the IR camera respectively have their corresponding sensor arrays. The sensor array of the RGB camera usually adopts a Bayer array, that is, a Bayer array in the RGGB format. The sensor array of the IR camera usually adopts an IR array. Compared with the single-eye RGB-IR camera in the related art, the RGB component and the IR component corresponding to the RGB camera and the IR camera in the binocular camera module 1 respectively occupy the full resolution of the RGB camera and the IR camera, and the binocular camera module 1 in the present disclosure can collect RGB image data and IR image data with higher resolution.

[0055] In one embodiment, if there are two binocular camera modules 1, then there are also two first data transmission units 2; and the signal output terminal of the ISP module of each binocular camera module 1 is connected to the signal input terminal of the corresponding first data transmission unit 2.

[0056] In one embodiment, the image data collected by the RGB camera and the IR camera is usually RAW format image data, and after being processed by the ISP module, the image data output by the ISP module is YUV format image data.

[0057] In one embodiment, HU4 is the core of the human-machine interaction unit of the vehicle, generally referring to the vehicle head unit and the central control screen, and the cockpit domain controller.

[0058] In one embodiment, after the ISP module processes the image data collected by the RGB camera and the IR camera and sends it to the SOC of the vehicle's HU4, the SOC presents the image corresponding to the processed image data on the human-machine interface of the HU4.

[0059] In one embodiment, the SOC can also send a diagnostic instruction to the binocular camera module 1 through the IIC protocol. The diagnostic instruction is used to diagnose whether the RGB camera or the IR camera in the binocular camera module 1 is working properly; the SOC can also receive the diagnostic result through the IIC protocol.

[0060] In one embodiment, the synchronous acquisition trigger signal can be provided by the micro control unit (MCU) in the HU4.

[0061] In one embodiment, taking the synchronous acquisition trigger signal provided by the MCU of the vehicle HU4 as an example:

[0062] When the vehicle is powered on, the HU4 powers on the RGB camera and the IR camera in the binocular camera module 1 through the MCU and generates a synchronous acquisition trigger signal, which is used to make the RGB camera and the IR camera in the binocular camera module 1 synchronously acquire the internal image data of the vehicle;

[0063] When the vehicle is in sleep mode, the HU4 powers off the RGB camera and the IR camera in the binocular camera module 1 through the MCU, and the RGB camera and the IR camera in the binocular camera module 1 do not work;

[0064] When the vehicle is awakened, the HU4 powers on the RGB camera and the IR camera in the binocular camera module 1 through the MCU, and the RGB camera and the IR camera in the binocular camera module 1 work;

[0065] When the vehicle is working, the HU4 powers on the RGB camera and the IR camera in the binocular camera module 1 through the MCU, and the RGB camera and the IR camera in the binocular camera module 1 work.

[0066] In one embodiment, the HU4 powers on or off the RGB camera and the IR camera in the binocular camera module 1 through the ISP module.

[0067] In one embodiment, if there are two or more binocular camera modules 1, then the HU4 powers on or off the RGB camera and the IR camera in each binocular camera module 1 through the ISP module.

[0068] In one embodiment, the MCU sends a synchronous acquisition trigger signal to the ISP module through the second data transmission unit 3 and the corresponding first data transmission unit 2, triggering the RGB camera and the IR camera in the binocular camera module 1 to synchronously expose.

[0069] In one embodiment, if there are multiple binocular camera modules 1, a synchronous acquisition trigger signal is sent to each RGB camera and each IR camera in the multiple binocular camera modules 1.

[0070] The data processing system provided by the present disclosure includes: at least one binocular camera module, the binocular camera module includes an RGB camera, an IR camera and an ISP module; both the RGB camera and the IR camera are communicatively connected to the ISP module; the RGB camera and the IR camera are used to synchronously acquire image data inside the vehicle in response to a synchronous acquisition trigger signal; the ISP module is used to process the image data acquired by the RGB camera and the IR camera; at least one first data transmission unit having the same number as the binocular camera modules, the signal output end of the ISP module of each binocular camera module is connected to the signal input end of the corresponding first data transmission unit; a second data transmission unit, multiple signal input ends of the second data transmission unit are respectively connected to the signal output ends of each first data transmission unit; the second data transmission unit is used to send the image data acquired by the RGB camera and the IR camera processed by the ISP module to the SOC of the in-vehicle host HU; HU, HU includes an MCU and an SOC; the synchronous signal output end of the MCU is connected to the synchronous signal input end of the second data transmission unit; the MCU is used to provide a synchronous acquisition trigger signal for the corresponding binocular camera module through the second data transmission unit and the corresponding first data transmission unit; the synchronous acquisition trigger signal is used to trigger each binocular camera module in at least one binocular camera module to synchronously expose; the signal input end of the SOC is connected to the signal output end of the second data transmission unit.

[0071] According to the solution of the present disclosure, the MCU sends a synchronous acquisition trigger signal to each binocular camera module to control each RGB camera and each IR camera in each binocular camera module to synchronously acquire image data inside the vehicle; the image data acquired by the RGB camera and the IR camera processed by the ISP module in each binocular camera module is sent to the SOC of the HU through the first data transmission unit and the second data transmission unit. The camera module in this solution uses a binocular camera module. Compared with the traditional monocular camera, the RGB components and IR components corresponding to the RGB camera and the IR camera in the binocular camera module respectively occupy the full resolution of the RGB camera and the IR camera, so that the binocular camera module can acquire RGB image data and IR image data with higher resolution; the RGB image data and IR image data with higher resolution are processed by the ISP module and then transmitted to the SOC through the first data transmission unit and the second data transmission unit, and the image data can display clearer images.

[0072] In one embodiment, the binocular camera module 1 further includes an IR fill light driving module and an IR fill light;

[0073] The signal input end of the IR fill light driving module is connected to the signal output end of the IR camera. The IR fill light driving module is used to send a driving signal to the IR fill light under the control of the enabling signal output by the IR camera; the driving signal is used to turn on the IR fill light;

[0074] The control end of the IR fill light is connected to the signal output end of the IR fill light driving module. The IR fill light is used to fill light for the IR camera.

[0075] In one embodiment, if there are multiple binocular camera modules 1, then each binocular camera module 1 includes an IR fill light driving module and an IR fill light.

[0076] In one embodiment, the IR fill light driving module is controlled by an OR gate circuit at the signal output end of the IR camera.

[0077] In one embodiment, the number of IR fill lights can be one, two or more. For example, if the number of IR fill lights is two, the IR fill lights can be arranged on the left and right sides of the IR camera.

[0078] In one embodiment, the number of IR fill lights can be increased or decreased according to the fill light requirement of the IR camera.

[0079] In one embodiment, the IR fill light can be an LED fill light or a laser.

[0080] In one embodiment, correspondingly, the IR fill light driving module can be an LED driving module or a laser driving module.

[0081] In one embodiment, under the control of the enable signal output by the IR camera, the IR fill light driving module sends a driving signal to the IR fill light to turn on the IR fill light, so as to fill light for the IR camera, and the clarity of the image data collected by the IR camera can be further improved.

[0082] In one embodiment, the first data transmission unit 2 includes:

[0083] A serializer group, which includes a first serializer and a second serializer;

[0084] A Fakra connector group, which includes a first Fakra connector and a second Fakra connector;

[0085] The signal input end of the first serializer is connected to the first signal output end of the ISP module of the corresponding binocular camera module 1, and the signal output end of the first serializer is connected to the signal input end of the corresponding first Fakra connector; the first serializer is used to transmit the image data collected by the RGB camera processed by the corresponding ISP module to the corresponding first Fakra connector;

[0086] The signal input end of the second serializer is connected to the second signal output end of the ISP module of the corresponding binocular camera module 1, and the signal output end of the second serializer is connected to the signal input end of the corresponding second Fakra connector; the second serializer is used to transmit the image data collected by the IR camera processed by the corresponding ISP module to the corresponding second Fakra connector.

[0087] In one embodiment, the first serializer is used to convert the parallel image data collected by the RGB camera processed by the ISP module into serial RGB image data.

[0088] In one embodiment, the second serializer is used to convert the parallel image data collected by the IR camera processed by the ISP module into serial IR image data.

[0089] In one embodiment, the YUV-formatted RGB image data and IR image data processed by the ISP module are transmitted to the first serializer and the second serializer respectively through the Mobile Industry Processor Interface (MIPI) protocol.

[0090] In one embodiment, the RGB image data and the IR image data arrive at the first serializer and the second serializer synchronously and in the same frame.

[0091] In one embodiment, the first Fakra connector is used to transmit the processed RGB image data in YUV format output by the first serializer.

[0092] In one embodiment, the second Fakra connector is used to transmit the processed IR image data in YUV format output by the second serializer.

[0093] In one embodiment, the first Fakra connector and the second Fakra connector respectively receive the processed RGB image data and IR image data in YUV format output by the first serializer and the second serializer through the second generation version of the Gigabit Multimedia Serial Link (GMSL) protocol, that is, the GMSL2 protocol.

[0094] In one embodiment, the second data transmission unit 3 includes:

[0095] A multi-in-one Fakra connector, and multiple signal input ends of the multi-in-one Fakra connector are respectively connected to the signal output ends of the first Fakra connector and the second Fakra connector of each first data transmission unit 2;

[0096] A deserializer, an image signal input end of the deserializer is connected to the signal output end of the multi-in-one Fakra connector, and an image signal output end of the deserializer is connected to the signal input end of the SOC.

[0097] In one embodiment, if there are two binocular camera modules 1, corresponding to two first data transmission units 2; then there are two first Fakra connectors and two second Fakra connectors, and a total of four signal output ends are connected to the input end of the multi-in-one Fakra connector. At this time, the multi-in-one Fakra connector is a four-in-one Fakra connector.

[0098] In one embodiment, the first Fakra connector and the second Fakra connector transmit the RGB image data and the IR image data to the multi-in-one Fakra connector through the GMSL2 protocol at a bandwidth transmission rate of 6Gbps.

[0099] In one embodiment, the deserializer is used to receive the RGB image data and the IR image data transmitted by the multi-in-one Fakra connector.

[0100] In one embodiment, the deserializer is used to convert the serial RGB image data and IR image data into parallel RGB image data and IR image data.

[0101] In one embodiment, the multi-in-one Fakra connector transmits the RGB image data and the IR image data to the deserializer through the GMSL2 protocol.

[0102] In one embodiment, receiving the image data of the RGB camera and the IR camera by using the multi - in - one Fakra connector can save the space of the data processing system.

[0103] In one embodiment, the SOC includes a diagnostic module;

[0104] The diagnostic module is communicatively connected to the GPIO3 port of the deserialiser of the second data transmission unit 3;

[0105] The diagnostic module is used to send diagnostic instructions to each binocular camera module 1 through the second data transmission unit 3 and the corresponding first data transmission unit 2, and determine whether each binocular camera module 1 is faulty according to the return value of the GPIO3 port of the deserialiser.

[0106] In one embodiment, the diagnostic module sends a diagnostic instruction to the RGB camera through the first serializer.

[0107] In one embodiment, the diagnostic module sends diagnostic instructions to the IR fill - light driving module and the IR camera through the second serializer.

[0108] In one embodiment, the status of the GPIO1 port of the first serializer and the second serializer of the first data transmission unit 2 can be transmitted to the GPIO3 port of the deserialiser.

[0109] In one embodiment, when the RGB camera or the IR camera has faults such as sensor freeze - frame, abnormal power supply, temperature warning, abnormal power chip, MIPI transmission anomaly, etc., the return value of the GPIO1 port of the first serializer or the second serializer will be pulled low to a low level.

[0110] In one embodiment, the deserialiser only leaves one GPIO3 port for the diagnosis of the RGB camera and the IR camera module. The SOC polls the GPIO3 port of the deserialiser. If the return value of the GPIO3 port of the deserialiser is low level, it means that the RGB camera or the IR camera is faulty.

[0111] In one embodiment, if the return value of the GPIO3 port of the deserialiser is low level, it is determined that at least one of the binocular camera modules 1 in the binocular camera module 1 is faulty; at this time, the diagnostic module is triggered to read the return values of the GPIO1 port and the GPIO2 port of the first serializer and the second serializer of each first data transmission unit 2 through the IIC protocol:

[0112] In one embodiment, if the return value of the GPIO1 port of the first serializer of the corresponding first data transmission unit 2 is low level, it is determined that the RGB camera in the corresponding binocular camera module 1 is faulty;

[0113] In one embodiment, if the return value of the GPIO2 port of the second serializer of the corresponding first data transmission unit 2 is low level, it is determined that the IR camera in the corresponding binocular camera module 1 is faulty;

[0114] In one embodiment, if the return values of both the GPIO1 port and the GPIO2 port of the second serializer of the corresponding first data transmission unit 2 are low level, it is determined that the IR fill light driving module in the corresponding binocular camera module 1 is faulty.

[0115] In one embodiment, diagnostic instructions are sent to the ISP module through the first data transmission unit 2 and the second data transmission unit 3 to diagnose whether the binocular camera module 1 is working properly, which can improve the reliability of the data processing system.

[0116] In one embodiment, the first serializer and the corresponding first Fakra connector, the second serializer and the corresponding second Fakra connector, between the first Fakra connector and the second Fakra connector and the multi - in - one Fakra connector, and between the multi - in - one Fakra connector and the deserializer are all connected through coaxial cables;

[0117] Between the ISP module in each binocular camera module 1 and the corresponding first serializer and second serializer, the image data processed by the ISP module is transmitted through the MIPI protocol.

[0118] In one embodiment, the coaxial cable consists of a central conductor (usually a copper wire), an insulating dielectric layer surrounding the central conductor, a shielding layer (usually a braided or foil - like metal layer), and an outer sheath. Compared with ordinary cables, the coaxial cable has strong anti - interference ability, good shielding performance, and stable data transmission.

[0119] In one embodiment, between the ISP module in each binocular camera module 1 and the corresponding first serializer, the RGB image data processed by the ISP module is transmitted through the MIPI protocol.

[0120] In one embodiment, between the ISP module in each binocular camera module 1 and the corresponding second serializer, the IR image data processed by the ISP module is transmitted through the MIPI protocol.

[0121] In one embodiment, between the first serializer and the corresponding first Fakra connector, the second serializer and the corresponding second Fakra connector, between the first Fakra connector and the second Fakra connector and the multi - in - one Fakra connector, and between the multi - in - one Fakra connector and the deserializer, the image data processed by the ISP module is transmitted through the GMSL2 protocol at a preset bandwidth transmission rate.

[0122] In one embodiment, according to the transmission requirements of RGB image data and IR image data, the size of the preset bandwidth transmission rate is determined. For example, if the size of the preset bandwidth transmission rate is 6 Gbps, it means that 6 G bits of RGB image data and IR image data are transmitted per second.

[0123] In one embodiment, HU4 further includes a power management unit;

[0124] The signal input end of the power management unit is connected to the power signal output end of the deserializer, and the power signal output end of the power management unit is connected to the signal input end of the multi - in - one Fakra connector; the power management unit is used to supply power to each binocular camera module 1 through the second data transmission unit 3 and the corresponding first data transmission unit 2.

[0125] In one embodiment, when the vehicle is powered on, through the power control signal output from the power signal output end of the deserializer, each binocular camera module 1 is powered on through the first data transmission unit 2 and the second data transmission unit 3.

[0126] In one embodiment, when the vehicle goes into sleep mode, through the power control signal output from the power signal output end of the deserializer, each binocular camera module 1 is powered off through the first data transmission unit 2 and the second data transmission unit 3.

[0127] In one embodiment, when the vehicle wakes up, through the power control signal output from the power signal output end of the deserializer, each binocular camera module 1 is powered on through the first data transmission unit 2 and the second data transmission unit.

[0128] In one embodiment, when the vehicle is working, through the power control signal output from the power signal output end of the deserializer, each binocular camera module 1 is powered on through the first data transmission unit 2 and the second data transmission unit 3.

[0129] In one embodiment, the RGB camera and the IR camera in each binocular camera module 1 respectively send the collected image data in RAW format to the corresponding ISP module. After the ISP module processes the image data collected by the RGB camera and the IR camera respectively, it outputs the image data in YUV format to the corresponding first serializer and second serializer.

[0130] In one embodiment, the RAW - format RGB image data collected by the RGB camera is processed by at least one of the following and the RGB image data in YUV format is output.

[0131] In one embodiment, gamma correction, white balance, color correction, contrast adjustment, saturation adjustment, or region - of - interest extraction.

[0132] In one embodiment, the RAW format IR image data collected by the IR camera is processed by at least one of the following processes, and YUV format IR image data is output.

[0133] In one embodiment, gray coefficient correction, sharpening adjustment, brightness uniformity adjustment, saturation adjustment, or region of interest extraction.

[0134] In one embodiment, in the YUV format, the Y channel represents the luminance signal in the image data, and the U channel and V channel represent the chrominance signals in the image data.

[0135] In one embodiment, in order to obtain better images, the technical parameters of the binocular camera module should also meet the following technical parameter requirements.

[0136] In one embodiment, the resolution of the IR camera is not less than 2 million, specifically it can be 1600*1300; the resolution of the RGB camera can be 2548*2072.

[0137] In one embodiment, the number of fill lights of the IR camera can be 2.

[0138] In one embodiment, the ISP module supports automatic exposure and version upgrade of the IR camera, etc.; the ISP module supports gray coefficient correction, sharpening adjustment, brightness uniformity adjustment, saturation adjustment, or region of interest extraction of the RGB camera, etc.

[0139] In one embodiment, the image output format of the IR camera is RAW8 or RAW10; the image output format of the RGB camera is RGB888 or YUV422, and the specific image output formats of the IR camera and the RGB camera can be adjusted according to design requirements.

[0140] In one embodiment, the ISP module is also integrated with diagnostic functions such as ISP module diagnosis, IR fill light driver module diagnosis, and power chip diagnosis.

[0141] In one embodiment, when the vehicle is driving in the following situations, the IR camera needs red exposure: when the vehicle is driving on the highway, when the vehicle is driving in a lightless basement, when the vehicle is driving in a road environment with lights at night; and the red exposure degrees of all the IR cameras included in the binocular camera module 1 on the vehicle are the same.

[0142] In one embodiment, the start-up time of the IR camera and the RGB camera in the binocular camera module 1 in response to the synchronous acquisition trigger signal does not exceed 200 milliseconds.

[0143] In one embodiment, the distance between the IR camera and the RGB camera is 55 mm to 65 mm, with an error not exceeding 0.5 mm, and the distance between the R camera and the RGB camera is preferably 58.5 mm.

[0144] In one embodiment, the hardware parameters of the IR camera and the RGB camera should also meet the following standards:

[0145] The diameter is 14 millimeters, with an error not exceeding 0.1 millimeter;

[0146] The length of the front shell is 144 millimeters, with an error not exceeding 0.3 millimeter. The length of the front shell refers to the length excluding the mounting ears;

[0147] The height of the front shell is 23 millimeters, with an error not exceeding 0.3 millimeter. The height of the front shell refers to the height excluding the connector;

[0148] The thickness is 30.8 millimeters, with an error not exceeding 0.5 millimeter. The thickness refers to the thickness from the camera lens to the most protruding point of the rear shell of the camera.

[0149] In one embodiment, the waterproof ratings of the IR camera and the RGB camera meet IP5K0 for side outlet and IP5K2 for back outlet.

[0150] In one embodiment, the RGB lens needs to be chamfered with R0.3.

[0151] In one embodiment, the angle between the optical axis of the binocular camera module 1 and the mating surface of the camera mounting ear is controlled within 1 degree.

[0152] In one embodiment, in the optical waveform output by the fill light of the IR camera, the total proportion of the rising edge and the falling edge is not greater than 20%, and the difference in the proportion of the peak waveform does not exceed 10%. The total proportion of the rising edge and the falling edge of the optical waveform not being greater than 20% indicates that the stability of the optical signal output by the fill light is better, which can ensure the normal operation of the IR camera. The difference in the proportion of the peak waveform of the optical waveform not exceeding 10% indicates that the consistency of the optical signal output by the fill light is better, thereby improving the consistency of the image data collected by the IR camera.

[0153] In one embodiment, the depth of field of the RGB camera and the IR camera in the binocular camera module is from 0.2 meters to 3 meters. Since the depth of field is used to characterize the range within which the camera can capture clear images, based on this, when arranging the binocular camera module, the closest distance of the lenses of the RGB camera and the IR camera from their preset acquisition range is 0.2 meters, and the farthest distance is 3 meters. For example, when the binocular camera module arranged in the front cockpit of the vehicle is used to collect the actions of the passengers in the front seats of the vehicle, at this time, the distance between the lens arrangement positions of the RGB camera and the IR camera and the front seats of the vehicle should be between 0.2 meters and 3 meters.

[0154] In one embodiment, the requirements for the gain and exposure time of the RGB camera and the IR camera in the binocular camera module should meet the following conditions:

[0155] When shooting a whiteboard target at an illuminance of 1000 lumens and a shooting distance of 1 meter, in the collected image data, the gray value is greater than 200;

[0156] When the illuminance is 1 lumen and the shooting distance is 1.2 meters, the signal-to-noise ratio is greater than 15 dB;

[0157] When the ambient noise is 30 dB and the shooting distance is 2 cm, the noise measured by the camera is not greater than 31 dB.

[0158] The greater the gain, the greater the magnification factor for amplifying the original image data collected by the camera. The longer the exposure time, the greater the gray value in the image data collected by the camera. At the same time, the gain and exposure time are also positively correlated with the signal-to-noise ratio. For example, when the illuminance is 1 lumen, which is relatively low, in order to collect higher-quality image data, it is necessary to increase the gain and exposure time of the camera, and at the same time, the signal-to-noise ratio of the camera is also increased.

[0159] In one embodiment, to meet the requirements of vehicle-related services, the frame rate of the image data collected by the RGB camera and the IR camera in the binocular camera module is not less than 30 fps.

[0160] In one embodiment, the calibration accuracy of the RGB camera and the IR camera in the binocular camera module should meet the following conditions:

[0161] When calibrating multiple collected image data, each calibrated image data is reprojected into the physical coordinates, and the average value of the pixel errors between the actual coordinates and the reprojected physical coordinates of the multiple image data is not greater than 0.3 pixel;

[0162] When calibrating a single collected image data, the calibrated image data is reprojected into the physical coordinates, and the average value of the pixel errors between the actual coordinates and the reprojected physical coordinates of the corresponding image data is not greater than 0.5 pixel;

[0163] When calibrating a single collected image data, after projecting the single image data into the world coordinate system and then inverse-projecting it into the physical coordinates, the average value of the pixel errors between the actual coordinates and the physical coordinates of the corresponding image data is not greater than 0.64 pixel.

[0164] In one embodiment, the depth accuracy of the RGB camera and the IR camera in the binocular camera module should meet the following conditions:

[0165] When shooting a flat target at a shooting distance of 1.1 meters, the depth error of the collected image data is not greater than 6 mm;

[0166] When shooting a flat target at a shooting distance of 1.1 meters, the difference between the depth value of the acquired image data calculated by the fitting algorithm and the actual depth value of the flat target is not greater than 1 centimeter.

[0167] The data processing system provided by the embodiments of the present disclosure includes: at least one binocular camera module, the binocular camera module includes an RGB camera, an IR camera, and an ISP module; both the RGB camera and the IR camera are communicatively connected to the ISP module; the RGB camera and the IR camera are used to synchronously acquire image data inside the vehicle in response to a synchronous acquisition trigger signal; the ISP module is used to process the image data acquired by the RGB camera and the IR camera; at least one first data transmission unit having the same number as the binocular camera modules, the signal output end of the ISP module of each binocular camera module is connected to the signal input end of the corresponding first data transmission unit; a second data transmission unit, multiple signal input ends of the second data transmission unit are respectively connected to the signal output ends of each first data transmission unit; the second data transmission unit is used to send the image data acquired by the RGB camera and the IR camera processed by the ISP module to the SOC of the in-vehicle host HU; HU, HU includes an MCU and an SOC; the synchronous signal output end of the MCU is connected to the synchronous signal input end of the second data transmission unit; the MCU is used to provide a synchronous acquisition trigger signal for the corresponding binocular camera module through the second data transmission unit and the corresponding first data transmission unit; the synchronous acquisition trigger signal is used to trigger each binocular camera module in at least one binocular camera module to synchronously expose; the signal input end of the SOC is connected to the signal output end of the second data transmission unit.

[0168] According to the solution of the present disclosure:

[0169] First, a synchronous acquisition trigger signal is sent to each binocular camera module through the MCU to control each RGB camera and each IR camera in each binocular camera module to synchronously acquire image data inside the vehicle; the image data acquired by the RGB camera and the IR camera processed by the ISP module in each binocular camera module is sent to the SOC of HU through the first data transmission unit and the second data transmission unit. In this solution, the camera module uses a binocular camera module. Compared with the traditional monocular camera, the RGB component and the IR component corresponding to the RGB camera and the IR camera in the binocular camera module respectively occupy the full resolution of the RGB camera and the IR camera, so that the binocular camera module can acquire higher-resolution RGB image data and IR image data; the higher-resolution RGB image data and IR image data are processed by the ISP module and then transmitted to the SOC through the first data transmission unit and the second data transmission unit, and the image data can show a clearer image.

[0170] Secondly, under the control of the enabling signal output by the IR camera, the IR fill light driving module sends a driving signal to the IR fill light. The driving signal is used to turn on the IR fill light, which can further improve the clarity of the image data collected by the IR camera.

[0171] Thirdly, receiving the image data of the RGB camera and the IR camera by using the multi-in-one Fakra connector can save the space of the data processing system.

[0172] Thirdly, sending a diagnostic instruction to the ISP module through the first data transmission unit and the second data transmission unit to diagnose whether the RGB camera or the IR camera is working properly can improve the reliability of the data processing system.

[0173] The present disclosure also provides a vehicle, including the data processing system according to any embodiment of the present disclosure;

[0174] Wherein, if the number of the binocular camera modules 1 in the data processing system is two, the two binocular camera modules 1 are respectively arranged at corresponding positions in the front row cockpit and the rear row cockpit of the vehicle;

[0175] The MCU is used to provide a synchronous acquisition trigger signal for the two binocular camera modules 1 through the second data transmission unit 3 and the corresponding first data transmission unit 2, so that the two binocular camera modules 1 are synchronously exposed.

[0176] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0177] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0178] In the present invention, unless otherwise clearly specified or limited, the terms "mounted", "connected", "coupled", "fixed", etc. shall be construed in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral body; it may be a mechanical connection, an electrical connection, or communication with each other; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the internal communication of two components or the interaction relationship between two components, unless otherwise clearly limited. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0179] In the present invention, unless otherwise clearly specified or limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may be that the first feature is directly above or obliquely above the second feature, or merely indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature being "under", "beneath" and "underneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely indicates that the horizontal height of the first feature is lower than that of the second feature.

[0180] In the present invention, the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0181] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A data processing system, characterized in that, Including: At least one binocular camera module, the binocular camera module includes a Red Green Blue (RGB) camera, an Infrared (IR) camera, and an Image Signal Processing (ISP) module; Both the RGB camera and the IR camera are communicatively connected to the ISP module; the RGB camera and the IR camera are used to synchronously collect image data inside the vehicle in response to a synchronous acquisition trigger signal; The ISP module is used to process the image data collected by the RGB camera and the IR camera; At least one first data transmission unit having the same number as the binocular camera modules, a signal output end of the ISP module of each binocular camera module is connected to a signal input end of the corresponding first data transmission unit; A second data transmission unit, multiple signal input ends of the second data transmission unit are respectively connected to signal output ends of each first data transmission unit; the second data transmission unit is used to send the image data collected by the RGB camera and the IR camera processed by the ISP module to a System-on-Chip (SOC) of an in-vehicle host unit (HU); The HU, the HU includes a Micro Control Unit (MCU) and a SOC; A synchronous signal output end of the MCU is connected to a synchronous signal input end of the second data transmission unit; the MCU is used to provide a synchronous acquisition trigger signal for the corresponding binocular camera module through the second data transmission unit and the corresponding first data transmission unit; The synchronous acquisition trigger signal is used to trigger synchronous exposure of each binocular camera module in at least one binocular camera module; A signal input end of the SOC is connected to a signal output end of the second data transmission unit.

2. The data processing system according to claim 1, wherein The binocular camera module further includes an IR fill light driving module and an IR fill light; A signal input end of the IR fill light driving module is connected to a signal output end of the IR camera, the IR fill light driving module is used to send a driving signal to the IR fill light under the control of an enabling signal output by the IR camera; the driving signal is used to turn on the IR fill light; A control end of the IR fill light is connected to a signal output end of the IR fill light driving module, the IR fill light is used to fill light for the IR camera.

3. The system according to claim 2, characterized in that, In the light waveform output by the fill light, the total proportion of the rising edge and the falling edge is not greater than 20%, and the proportion difference of the peak waveforms does not exceed 10%.

4. The data processing system according to claim 1, wherein The first data transmission unit includes: A serializer group, the serializer group includes a first serializer and a second serializer; A Fakra connector group, the Fakra connector group includes a first Fakra connector and a second Fakra connector; A signal input end of the first serializer is connected to a first signal output end of the ISP module of the corresponding binocular camera module, a signal output end of the first serializer is connected to a signal input end of the corresponding first Fakra connector; the first serializer is used to transmit the image data collected by the RGB camera processed by the corresponding ISP module to the corresponding first Fakra connector; The signal input end of the second serializer is connected to the second signal output end of the ISP module of the corresponding binocular camera module, and the signal output end of the second serializer is connected to the signal input end of the corresponding second Fakra connector; the second serializer is configured to transmit the image data collected by the IR camera processed by the corresponding ISP module to the corresponding second Fakra connector.

5. The data processing system according to claim 4, wherein The second data transmission unit includes: A multi-in-one Fakra connector, and multiple signal input ends of the multi-in-one Fakra connector are respectively connected to the signal output ends of the first Fakra connector and the second Fakra connector of each of the first data transmission units; A deserialiser, the image signal input end of the deserialiser is connected to the signal output end of the multi-in-one Fakra connector, and the image signal output end of the deserialiser is connected to the signal input end of the SOC.

6. The data processing system according to claim 5, wherein The SOC includes a diagnostic module; The diagnostic module is communicatively connected to the GPIO3 port of the deserialiser of the second data transmission unit; The diagnostic module is configured to send diagnostic instructions to each binocular camera module through the second data transmission unit and the corresponding first data transmission unit, and determine whether each binocular camera module is faulty according to the return value of the GPIO3 port of the deserialiser.

7. The data processing system according to claim 5, wherein Between the first serializer and the corresponding first Fakra connector, between the second serializer and the corresponding second Fakra connector, between the first Fakra connector and the second Fakra connector and the multi-in-one Fakra connector, and between the multi-in-one Fakra connector and the deserialiser are all connected by coaxial cables; Between the ISP module in each binocular camera module and the corresponding first serializer and the second serializer, the image data processed by the ISP module is transmitted through the MIPI protocol.

8. The data processing system according to claim 5, wherein Between the first serializer and the corresponding first Fakra connector, between the second serializer and the corresponding second Fakra connector, between the first Fakra connector and the second Fakra connector and the multi-in-one Fakra connector, and between the multi-in-one Fakra connector and the deserialiser are all transmitted through the GMSL2 protocol at a preset bandwidth transmission rate for the image data processed by the ISP module.

9. The data processing system according to claim 5, wherein The HU further includes a power management unit; The signal input end of the power management unit is connected to the power signal output end of the deserialiser, and the power signal output end of the power management unit is connected to the signal input end of the multi-in-one Fakra connector; The power management unit is configured to supply power to each binocular camera module through the second data transmission unit and the corresponding first data transmission unit.

10. The data processing system according to any one of claims 5 to 9, characterized in that in each of the binocular camera modules, the RGB camera and the IR camera respectively send the acquired image data in RAW format to the corresponding ISP module, and after the ISP module processes the image data acquired by the RGB camera and the IR camera respectively, it outputs the image data in YUV format to the corresponding first serializer and the second serializer.

11. The data processing system according to claim 1, wherein The depth of field of the RGB camera and the IR camera in the binocular camera module is from 0.2 meters to 3 meters.

12. The data processing system according to claim 1, wherein The frame rate of the RGB camera and the IR camera in the binocular camera module for acquiring image data is not less than 30 fps.

13. A vehicle, characterized in that, Comprising the data processing system according to any one of claims 1 to 12; wherein, if the number of the binocular camera modules in the data processing system is two, the two binocular camera modules are respectively arranged at corresponding positions in the front row cockpit and the rear row cockpit of the vehicle; the MCU is configured to provide a synchronous acquisition trigger signal for the two binocular camera modules through the second data transmission unit and the corresponding first data transmission unit, so that the two binocular camera modules are synchronously exposed.

Images