Vehicle-mounted cockpit sensing equipment and its control method

Abstract

This invention discloses an in-vehicle cockpit sensing device and its control method. The in-vehicle cockpit sensing device integrates an image acquisition unit, an image sensor, a visual perception processing chip, and an in-vehicle processor. The image acquisition unit acquires images to be processed; the input of the image sensor is connected to the output of the image acquisition unit to receive the images to be processed and convert them into image signals; the input of the visual perception processing chip is connected to the output of the image sensor to receive the image signals to be processed and perform visual perception processing on them to obtain a perception result; the input of the in-vehicle processor is connected to the output of the visual perception processing chip to receive the perception result. This invention solves the technical problem in related technologies where integrating the cockpit visual perception system separately into the vehicle's infotainment system or ECU increases the overall vehicle hardware cost and increases the complexity of vehicle installation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle perception, in particular to an in-vehicle cabin perception device and a control method thereof. BACKGROUND

[0002] In the related art, with the continuous popularization of automobile intelligent technology, an intelligent cabin is arranged in the automobile, the automobile intelligent cabin realizes intelligent interaction of people, roads and vehicles through linkage of intelligent cabin interior and cabin electronics, and is an intelligent vehicle product for redefining the relationship between people and vehicles. The automobile cockpit is the most intuitive carrier of automobile technology, and with the continuous strengthening of personalized consumption and high-end consumption trends, the cabin electronics represented by intelligence and technology can meet the demand of consumers for intelligent driving configuration.

[0003] In-cabin monitoring is a new application that has emerged in recent years, which improves the safety of driving and passengers by monitoring the entire vehicle environment in real time. There are usually two ways of current in-cabin monitoring: the first way is to integrate the in-cabin visual perception system in the car machine system; the second way is to integrate the in-cabin visual perception system in a dedicated ECU, but these two integration ways have obvious disadvantages: for the first way, not only the computing capability of the car machine chip is required to be high, but also the hardware cost of the car machine is increased, and the visual perception system of each vehicle model of the car machine needs to be customized and developed; and for the second way, the hardware cost of the whole vehicle is also increased, and the separate setting of the ECU increases the complexity of the whole vehicle installation and wire harness arrangement.

[0004] At present, no effective solution has been proposed for the above problems. SUMMARY

[0005] The embodiments of the present application provide an in-vehicle cabin perception device and a control method thereof, a storage medium and an electronic device, to at least solve the technical problem that the in-cabin visual perception system is separately integrated in the car machine system or the ECU in the related art, which increases the hardware cost of the whole vehicle and the complexity of the whole vehicle installation.

[0006] According to one aspect of the present invention, an in-vehicle cockpit sensing device is provided, the in-vehicle cockpit sensing device integrating an image acquisition device, an image sensor, a visual perception processing chip, and an in-vehicle processor, wherein the image acquisition device is used to acquire an image to be processed; the input end of the image sensor is connected to the output end of the image acquisition device, and is used to receive the image to be processed and convert the image to be processed into an image signal to be processed; the input end of the visual perception processing chip is connected to the output end of the image sensor, and is used to receive the image signal to be processed and perform visual perception processing on the image signal to be processed to obtain a perception result; the input end of the in-vehicle processor is connected to the output end of the visual perception processing chip, and is used to receive the perception result.

[0007] Optionally, the step of performing visual perception processing on the image signal to be processed to obtain a perception result includes: reading pre-stored object feature points; extracting image feature points from the image signal to be processed; and comparing the object feature points with the image feature points to obtain the perception result.

[0008] Optionally, before extracting the image feature points of the image signal to be processed, the method further includes: performing image signal processing debugging on the image signal to be processed.

[0009] Optionally, the in-vehicle cockpit sensing device further integrates a memory, wherein the memory is connected to the visual perception processing chip and is used to store the object feature points.

[0010] Optionally, the in-vehicle cockpit sensing device further integrates one or more in-vehicle modules, wherein the input end of the one or more in-vehicle modules is connected to the output end of the in-vehicle processor for receiving the sensing results transmitted by the in-vehicle processor.

[0011] Optionally, the in-vehicle cockpit sensing device further integrates a serializer, wherein the input terminal of the serializer is connected to the output terminal of the visual perception processing chip, and the output terminal of the serializer is connected to the input terminal of the one or more in-vehicle modules, for receiving the image signal to be processed and outputting the image signal to be processed to the one or more in-vehicle modules.

[0012] Optionally, the in-vehicle cockpit sensing device also integrates a power supply, wherein the output terminal of the power supply is connected to the input terminal of the visual perception processing chip for supplying power to the visual perception processing chip.

[0013] Optionally, the vehicle-mounted cockpit sensing device also integrates a power converter, wherein the input terminal of the power converter is connected to the output terminal of the power supply, and the output terminal of the power converter is connected to the input terminals of the image acquisition device and the image sensor, for converting the power supply to power the image acquisition device and the image sensor.

[0014] Optionally, the vehicle-mounted processor includes: a system fault register for monitoring the operating status of the vision perception processing chip, and updating and storing the vision perception processing chip fault information if the vision perception processing chip malfunctions; and a power fault register for monitoring the operating status of the power supply, and updating and storing the power supply fault information if the power supply malfunctions.

[0015] Optionally, the vehicle processor sends the visual perception processing chip fault information and the power supply fault information to the one or more vehicle modules according to a fault diagnosis protocol.

[0016] According to another aspect of the present invention, a control method for an in-vehicle cockpit sensing device is also provided, comprising: acquiring an image to be processed; integrating the image to be processed into an image signal to be processed; performing visual perception processing on the image signal to be processed to obtain a perception result; and transmitting the perception result to an in-vehicle processor.

[0017] Optionally, the control method for the in-vehicle cockpit sensing device further includes: receiving a service upgrade request initiated by one or more in-vehicle modules; responding to the service upgrade request by sending a service upgrade package to the in-vehicle processor; the in-vehicle processor parsing the service upgrade package and storing a portion of the service upgrade firmware; the in-vehicle processor sending the service upgrade firmware of the visual perception processing chip to the visual perception processing chip; the visual perception processing chip saving the service upgrade firmware in a memory; the visual perception processing chip performing a service upgrade and, after the upgrade is completed, sending a first upgrade result to the in-vehicle processor; and the in-vehicle processor performing a service upgrade and, after the upgrade is completed, sending a second upgrade result to the one or more in-vehicle modules.

[0018] In this invention, an image sensor converts the image to be processed acquired by an image acquisition device into an image signal, which is then transmitted to a visual perception processing chip. The chip performs visual perception processing on the image signal to obtain a perception result, which is then transmitted to an on-board processor. The on-board processor then transmits the perception result to other on-board modules. This invention integrates image acquisition and image processing functions into an on-board cockpit perception device. While achieving cockpit perception and monitoring, it also supports fault diagnosis and remote system upgrades. Furthermore, it boasts strong vehicle model adaptability and multi-platform compatibility, reducing overall vehicle hardware costs. This solves the technical problem in related technologies where the cockpit visual perception system is separately integrated into the vehicle's infotainment system or ECU, increasing overall vehicle hardware costs and complicating vehicle installation. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a hardware block diagram of an optional vehicle-mounted cockpit sensing device according to an embodiment of the present invention;

[0021] Figure 2 This is a hardware block diagram of another optional vehicle-mounted cockpit sensing device according to an embodiment of the present invention;

[0022] Figure 3 This is a flowchart of an optional control method for an in-vehicle cockpit sensing device according to an embodiment of the present invention;

[0023] Figure 4 This is a hardware block diagram of an optional in-vehicle DMS system according to an embodiment of the present invention;

[0024] Figure 5 This is a hardware block diagram of an optional in-vehicle OMS system according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments are explained below:

[0028] Cockpit monitoring mainly includes DMS and OMS systems.

[0029] A Driver Monitor System (DMS) is a system that monitors a driver's fatigue level and dangerous driving behaviors around the clock. DMS functions include fatigue detection (e.g., yawning, closing eyes), dangerous action detection (smoking, making phone calls), distraction detection, and gesture recognition.

[0030] An Occupancy Monitoring System (OMS) is a system that detects sensory data from other passengers in the cabin to enhance safety. Based on the continuous integration of computer vision, deep learning, and artificial intelligence technologies, cabin monitoring is also an essential component of complete autonomous driving capabilities. OMS functions include rear-seat object detection, pet detection, and child detection.

[0031] Low-Voltage Differential Signaling (LVDS) is a differential signaling technology characterized by low power consumption, low bit error rate, low crosstalk, and low radiation. Its core feature is the use of extremely low voltage swing for high-speed differential data transmission, enabling point-to-point or point-to-multipoint connections. The transmission medium can be copper PCB traces or balanced cables.

[0032] Adaptive Cruise Control, or ACC for short.

[0033] A Universal Asynchronous Receiver / Transmitter (UART) converts data to be transmitted between serial and parallel communication.

[0034] The I2C bus is a simple, bidirectional, two-wire synchronous serial bus that requires only two wires to transmit information between devices connected to the bus.

[0035] The Mobile Industry Processor Interface (MIPI) is an open standard and specification for mobile application processors, initiated by the MIPI Alliance.

[0036] Serial Peripheral Interface (SPI) is a high-speed, full-duplex, synchronous communication bus.

[0037] Controller Area Network, abbreviated as CAN.

[0038] Over-the-Air (OTA) technology enables remote software management via mobile communication interfaces. Microcontroller Unit (MCU)

[0039] This application proposes an in-vehicle intelligent cockpit visual perception device, which can realize one or more of the following functions: driver face recognition, DMS (Driver Management System), and OMS (Operational Management System). Simultaneously, the device meets functional safety requirements such as real-time fault diagnosis and can perform OTA (Over-The-Air) upgrades via the CAN bus. The following detailed description, in conjunction with various embodiments, further illustrates this application.

[0040] Example 1

[0041] Figure 1 This is a hardware block diagram of an optional in-vehicle cockpit sensing device according to an embodiment of the present invention, such as... Figure 1 As shown, the vehicle-mounted cockpit sensing device 100 integrates an image acquisition device 101, an image sensor 102, a visual perception processing chip 103, and a vehicle-mounted processor 104.

[0042] In some embodiments, the vehicle-mounted cockpit sensing device is a camera, and the image acquisition device 101 is the lens of the camera.

[0043] The image acquisition device 101 is used to acquire images to be processed.

[0044] In this embodiment, the image acquisition device 101 can perform self-testing and initialize the image sensor 102 after power-on.

[0045] The images acquired by the image acquisition device 101 in this embodiment include, but are not limited to, RGB images, grayscale images, and depth images, based on the needs of various types of vehicles (the vehicle types in this embodiment include, but are not limited to, cars (such as pure electric vehicles, hybrid vehicles, and fuel vehicles), trucks, commercial vehicles, buses, etc.).

[0046] The input terminal of the image sensor 102 is connected to the output terminal of the image acquisition device 101. The image acquisition device 101 transmits the image to be processed to the image sensor 102, and the image sensor 102 is used to convert the image to be processed into an image signal.

[0047] In this embodiment, the image sensor 102 supports multiple automotive-grade sensors with 100W and 200W pixels from manufacturers including, but not limited to, OV and ON.

[0048] The input terminal of the visual perception processing chip 103 is connected to the output terminal of the image sensor 102. The image sensor 102 transmits the image signal to be processed to the visual perception processing chip 103. The visual perception processing chip 103 performs visual perception processing on the image signal to be processed and obtains the perception result.

[0049] In some embodiments, the input terminal of the visual perception processing chip 103 and the output terminal of the image sensor 102 are connected via a first type of ribbon cable. The first type of ribbon cable includes, but is not limited to, an FPC (Flexible Printed Board) ribbon cable, which includes, but is not limited to, MIPI transmission lines, clock signal lines, control signal lines, and I2C lines.

[0050] Optionally, in this embodiment, the image sensor 102 can also output clock signals and control signals to the visual perception processing chip 103 via a first type of ribbon cable.

[0051] The input terminal of the vehicle processor 104 is connected to the output terminal of the visual perception processing chip 103, and the vehicle processor 104 receives the perception results transmitted by the visual perception processing chip 103.

[0052] In this first embodiment, the input terminal of the vehicle processor 104 and the output terminal of the visual perception processing chip 103 are connected through a first type of bus, which includes, but is not limited to, the UART bus.

[0053] Example 2

[0054] Figure 2 This is a hardware block diagram of another optional vehicle-mounted cockpit sensing device according to an embodiment of the present invention, such as...Figure 2 As shown, the vehicle-mounted cockpit sensing device 200 integrates an image acquisition device 201, an image sensor 202, a visual perception processing chip 203, and an on-board processor 204. The image acquisition device 201 is used to acquire images to be processed.

[0055] The input terminal of the image sensor 202 is connected to the output terminal of the image acquisition device 201. The image acquisition device 201 transmits the image to be processed to the image sensor 202, and the image sensor 202 converts the image to be processed into an image signal.

[0056] The input terminal of the visual perception processing chip 203 is connected to the output terminal of the image sensor 202. The image sensor 102 transmits the image signal to be processed to the visual perception processing chip 203, and the visual perception processing chip 203 performs visual perception processing on the image signal to be processed to obtain the perception result.

[0057] The input terminal of the vehicle processor 204 is connected to the output terminal of the visual perception processing chip 203, and the vehicle processor 204 receives the perception results transmitted by the visual perception processing chip 203.

[0058] In this second embodiment, the visual perception processing chip 203 performs visual perception processing on the image signal to be processed to obtain a perception result, including: reading pre-stored object feature points; extracting image feature points from the image signal to be processed; and comparing the object feature points with the image feature points to obtain a perception result.

[0059] Optionally, before extracting image feature points from the image signal to be processed, the method further includes: image signal processing debugging of the image signal to be processed. This image signal processing debugging mainly refers to the visual perception processing chip 203 performing image signal processing (ISP) debugging on the image to be processed, obtaining a debugged image, and then performing visual perception algorithm processing on the debugged image to obtain the perception result. The visual perception algorithm processing is performed in the neural network processing units (NPU) of the visual perception processing chip 203.

[0060] Optionally, the vehicle-mounted cockpit sensing device 200 also integrates one or more vehicle-mounted modules, wherein the input terminals of the one or more vehicle-mounted modules are connected to the output terminals of the vehicle-mounted processor 204 for receiving the sensing results transmitted by the vehicle-mounted processor.

[0061] It should be noted that the vehicle module in this embodiment 2 can be, but is not limited to, a vehicle infotainment system or an ECU (Electronic Control Unit).

[0062] Alternatively, the in-vehicle cockpit sensing device 200 may also integrate a serializer 205, wherein the input of the serializer 205 is connected to the output of the visual perception processing chip 203 for receiving the image signal to be processed, and the output of the serializer is connected to the input of one or more in-vehicle modules for outputting the image signal to be processed to one or more in-vehicle modules.

[0063] It should be noted that in Embodiment 2, the connection between the serializer 205 and the visual perception processing chip 203 can be achieved through various interfaces, such as the MIPI interface.

[0064] Optionally, in this second embodiment, the serializer 205 can be connected to one or more vehicle modules via a second type of bus, and the vehicle processor 204 can be connected to one or more vehicle modules via a third type of bus. Optionally, the second type of bus can be represented as an LVDS bus, and the third type of bus can be represented as a CAN bus.

[0065] Optionally, the in-vehicle cockpit sensing device also integrates a memory 206, which is connected to the visual perception processing chip 203 and is used to store object feature points. In this embodiment, the types of serializer 205 and memory 206 illustrated can be selected arbitrarily. For example, serializer 205 can use various models from TI and Maxim, with MIPI interfaces.

[0066] In this embodiment, the vehicle-mounted cockpit sensing device 200 also integrates a power supply 207, wherein the output terminal of the power supply 207 is connected to the input terminal of the visual perception processing chip 203, and is used to supply power to the visual perception processing chip 203.

[0067] Optionally, the vehicle-mounted cockpit sensing device 200 also integrates a power converter 208, wherein the input terminal of the power converter 208 is connected to the output terminal of the power supply 207, and the output terminal of the power converter 208 is connected to the input terminals of the image acquisition device 201 and the image sensor 202, for converting the power output from the power supply 207 to power the image acquisition device 201 and the image sensor 202.

[0068] Optionally, the on-board processor 204 is connected to the ignition switch via a fourth type bus, which can be an ACC bus, i.e., an adaptive cruise control bus.

[0069] As an optional implementation of this embodiment, the vehicle-mounted cockpit sensing device 200 also integrates: a system fault register 210, used to monitor the working status of the visual perception processing chip 203; if the visual perception processing chip 203 malfunctions, the system updates and stores the visual perception chip fault information; and a power fault register 211, used to monitor the working status of the power supply 207; if the power supply 207 malfunctions, the power supply fault information is updated and stored. In this embodiment, the power fault register 211 can analyze the working status of the power supply 207 according to a predetermined time period, and based on the working status, indicate whether the power supply 207 has malfunctioned, generate power fault information, and display the power fault information.

[0070] The aforementioned system fault register 210 and power fault register 211 can be set in the vehicle processor 204.

[0071] In this embodiment, when the vehicle processor 204 receives a fault alarm from the visual perception processing chip 203, it updates the information in the system fault register stored internally by the vehicle processor 204. Simultaneously, the vehicle processor 204 periodically monitors the operating status of the power supply 207, and upon detecting a power supply fault, updates the information in the power fault register 211 stored internally by the vehicle processor 204.

[0072] Alternatively, in this embodiment, the vehicle processor 204 can send visual perception chip fault information and power supply fault information to one or more vehicle modules via a third-type bus according to a fault diagnosis protocol.

[0073] In this embodiment, by integrating the image acquisition device 201 and the visual perception processing chip 203 into the vehicle cockpit perception device 200, cockpit perception and monitoring are achieved, while supporting fault diagnosis and remote system upgrades. At the same time, it has strong vehicle adaptability and multi-platform compatibility, reducing the overall vehicle hardware cost. This solves the technical problem in related technologies where the cockpit visual perception system is separately integrated into the vehicle system or ECU, which increases the overall vehicle hardware cost and makes the overall vehicle installation complex.

[0074] According to an embodiment of the present invention, a control method embodiment for an in-vehicle cockpit sensing device is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0075] Figure 3 This is a flowchart of an optional control method for an in-vehicle cockpit sensing device according to an embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps:

[0076] Step S302: Acquire the image to be processed;

[0077] Step S304: Integrate the images to be processed into an image signal to be processed;

[0078] Step S306: Perform visual perception processing on the image signal to be processed to obtain the perception result;

[0079] Step S308: The perception results are transmitted to the vehicle processor.

[0080] Through the above steps, an image to be processed can first be acquired by an image acquisition device, then the image to be processed can be integrated into a processed image signal by an image sensor, and the processed image signal can be processed by a visual perception processing chip to obtain a perception result, which is then transmitted to the vehicle processor. In this embodiment, this application integrates image acquisition and visual perception processing functions into the vehicle cockpit perception device. While realizing cockpit perception and monitoring, it also supports fault diagnosis and remote system upgrades. At the same time, it has strong vehicle model adaptability and multi-platform compatibility, reducing the overall vehicle hardware cost. This solves the technical problem in related technologies where the cockpit visual perception system is separately integrated into the vehicle system or ECU, increasing the overall vehicle hardware cost and increasing the complexity of vehicle installation.

[0081] Optionally, the steps of acquiring images to be processed include: powering on the vehicle-mounted image acquisition device and initializing the image sensor of the vehicle-mounted cockpit sensing device; after the image sensor has completed initialization, controlling the vehicle-mounted image acquisition device to acquire images of a predetermined space area inside the vehicle.

[0082] Alternatively, the step of performing visual perception processing on the image signal to be processed to obtain a perception result includes: reading object feature points pre-stored in the memory through a visual perception processing chip; extracting image feature points from the image signal to be processed through a visual perception processing chip; and comparing the object feature points with the image feature points to obtain a perception result.

[0083] In this embodiment, object feature points may include facial feature points of the user object, such as facial feature points, prominent facial feature points, etc.

[0084] In this embodiment of the invention, the control method further includes: receiving a service upgrade request initiated by one or more vehicle modules; responding to the service upgrade request by sending a service upgrade package to the vehicle processor; the vehicle processor parsing the service upgrade package and storing a portion of the service firmware; the vehicle processor sending the service firmware of the vision perception chip to the vision perception processing chip; the vision perception processing chip saving the service firmware to a memory; the vision perception processing chip performing a service upgrade and, after the upgrade is completed, sending a first upgrade result to the vehicle processor; and the vehicle processor performing a preset service upgrade and, after the upgrade is completed, sending a second upgrade result to one or more vehicle modules.

[0085] The invention will now be described in conjunction with another alternative embodiment.

[0086] Example 3

[0087] The invention will now be described in detail with reference to two systems in an in-vehicle intelligent cockpit (DMS system and OMS system). In this embodiment, the image sensor and image acquisition device are integrated into a single sensing module.

[0088] The first system is the DMS system. In this embodiment, the in-vehicle cockpit sensing equipment is integrated into the DMS system. The DMS system can realize driver facial recognition and DMS functions, including but not limited to: fatigue detection (yawning, closing eyes), dangerous action detection (smoking, making phone calls), distraction detection, and gesture recognition.

[0089] Figure 4 This is a hardware block diagram of an optional in-vehicle DMS system according to an embodiment of the present invention, such as... Figure 4 As shown, the system includes: a DMS system sensing module and a DMS vision perception processing board, wherein,

[0090] The DMS system sensing module includes at least an image acquisition device and an image sensor. After the image acquisition device acquires the image to be processed, the image sensor integrates the acquired image to be processed into an image signal to be processed and transmits it to the DMS vision perception processing board.

[0091] In this embodiment, the image acquisition device can be the lens of the DMS camera, and the DMS system sensing module and the DMS visual perception processing board are integrated in the DMS camera.

[0092] The DMS vision perception processing board includes: a vision perception processing chip, an on-board processor, a serializer, a memory, and a power supply. The vision perception processing chip is connected to the image sensor in the DMS system sensing module via an FPC cable, to the serializer via a MIPI line, to the on-board processor via a UART bus, and to the memory via a serial peripheral interface SPI.

[0093] After receiving the image signal to be processed, the visual perception processing chip performs visual perception processing on the image signal to obtain the perception result, and transmits the perception result to the vehicle processor via the UART bus. The vehicle processor then transmits the perception result to the vehicle's infotainment system or other vehicle ECUs via the CAN bus.

[0094] exist Figure 4 In the process, the serializer is connected to the vehicle's infotainment system or ECU via the LVDS bus, and the ignition switch is connected to the vehicle's processor via the ACC bus controlled by the key.

[0095] Figure 4 In this system, power is supplied to the DMS vision perception processing board via automotive power supply. The power supply powers the vision perception processing chip. At the same time, the power in the power supply is converted by the power converter to power the various modules (image sensor and image acquisition device) in the DMS system sensing module.

[0096] The aforementioned in-vehicle DMS system consists of two parts: the DMS vision perception processing board and the DMS system sensing module. The DMS vision perception processing board includes a vision perception processing chip, an in-vehicle processor MCU, a serializer, memory, and a power supply. The DMS system sensing module includes an image sensor and an image acquisition device. The DMS system sensing module transmits the MIPI signal, RST control signal, and MCLK clock signal of the transmitted image to the DMS vision perception processing board via an FPC cable. After receiving the image data from the DMS system sensing module, the DMS vision perception processing board processes the image using a vision perception algorithm on the vision perception processing chip to obtain the perception result. This result is then transmitted to the in-vehicle processor via the UART bus. The in-vehicle processor then transmits the perception result to the vehicle's infotainment system or other in-vehicle ECUs via the CAN bus.

[0097] Meanwhile, in this embodiment, the visual perception processing chip transmits the image to the serializer via MIPI, and the serializer transmits the image to the vehicle's infotainment system or other vehicle ECU via LVDS.

[0098] The following is a schematic illustration of the workflow of the hardware system of the vehicle-mounted DMS system. The workflow includes:

[0099] 1. The image acquisition device is powered on and begins self-testing. At the same time, the image sensor is initialized.

[0100] 2. After the image sensor completes initialization, control the image acquisition device to acquire images of a predetermined space area inside the vehicle;

[0101] 3. The MIPI signal, RST control signal and MCLK clock signal of the transmitted image are transmitted to the DMS visual perception processing board through the FPC cable. The visual perception processing chip performs ISP (Image Signal Processing) debugging on the image.

[0102] 4. Process the debugged image using a visual perception algorithm to obtain the perception result;

[0103] 5. The visual perception processing chip transmits the perception results to the vehicle processor via the UART bus. At the same time, it outputs the image to the serializer via MIPI, and then the serializer outputs the image to the engine or other vehicle ECUs via LVDS.

[0104] 6. After receiving the sensing results, the on-board processor transmits the sensing results to the vehicle's infotainment system or other on-board ECUs via the CAN bus.

[0105] The following is an illustrative explanation of the face registration and face recognition process in the DMS system.

[0106] Before using the facial recognition function of the in-vehicle DMS system, users need to register their faces. Users can upload their facial images to the system by following the system prompts or by actively uploading or inputting their own information. The system will then recognize the facial images, obtain facial features (including facial features, features of various points on the face, etc.), and save the facial features to the storage.

[0107] After completing face registration, users can use the system's face recognition function. For users who need face recognition, the face image collected by the DMS system's sensing module can be transmitted to the DMS visual perception processing board. The visual perception processing chip extracts facial features and compares the extracted features with the facial features stored in the memory. The comparison result, i.e., the perception result, is sent to the vehicle processor. The vehicle processor can then send the perception result to the vehicle's infotainment system or other vehicle ECUs via the CAN protocol.

[0108] The following is an illustrative explanation of the fault diagnosis operation of the DMS system.

[0109] 1. The visual perception processing chip periodically monitors the working status of each hardware component in the DMS system. If the working status of any hardware component indicates a malfunction, a fault alarm is sent to the vehicle processor via UART.

[0110] 2. After receiving a fault alarm, the on-board processor updates and stores the visual perception chip fault information in the system fault register.

[0111] Meanwhile, the vehicle processor periodically monitors the power supply's operating status. If the power supply's operating status indicates a power supply failure, the power supply fault register in the vehicle processor is updated and the power supply fault information is stored.

[0112] 3. The on-board processor sends the visual perception chip fault information and power supply fault information to the vehicle's infotainment system or other on-board ECUs via the CAN bus according to the fault diagnosis protocol.

[0113] The following is an illustrative explanation of OTA upgrades for the DMS system.

[0114] 1. The vehicle's infotainment system or other vehicle ECUs initiate an OTA upgrade request and send the OTA upgrade package to the vehicle's processor via the CAN bus;

[0115] 2. The vehicle processor parses the OTA upgrade package and stores part of the OTA firmware in the internal storage of the vehicle processor;

[0116] 3. The onboard processor sends the OTA firmware of the vision perception processing chip to the vision perception processing chip;

[0117] 4. The visual perception processing chip saves its OTA firmware to the memory;

[0118] 5. The visual perception processing chip performs an OTA upgrade. After the upgrade is completed, the upgrade result (i.e., the first upgrade result) is sent to the vehicle processor via the UART bus.

[0119] 6. The vehicle processor performs an OTA upgrade. After the upgrade is completed, the upgrade result (i.e., the second upgrade result) is sent to the vehicle's infotainment system or other vehicle ECUs.

[0120] The second system is the OMS system. In this embodiment, the in-vehicle cabin sensing equipment is integrated into the OMS system. The OMS system can realize OMS functions, including but not limited to: rear seat object detection, pet detection, and child detection.

[0121] Figure 5 This is a hardware block diagram of an optional in-vehicle OMS system according to an embodiment of the present invention, such as... Figure 5 As shown, the OMS system includes: an OMS system sensing module and an OMS vision perception processing board, wherein,

[0122] The OMS system sensing module includes at least an image acquisition device and an image sensor. After the image acquisition device acquires the image to be processed, the image sensor integrates the acquired image to be processed into an image signal to be processed and transmits it to the OMS vision perception processing board.

[0123] In this embodiment, the image acquisition device can be the lens of the OMS camera, and the OMS system sensing module and the OMS vision perception processing board are integrated in the OMS camera.

[0124] The OMS vision perception processing board includes: a vision perception processing chip, an on-board processor, a serializer, a memory, and a power supply. The vision perception processing chip is connected to the image sensor in the OMS system sensing module via an FPC cable, to the serializer via a MIPI line, to the on-board processor via a UART bus, and to the memory via a serial peripheral interface SPI.

[0125] After receiving the image signal to be processed, the visual perception processing chip performs visual perception processing on the image signal to obtain the perception result, and transmits the perception result to the vehicle processor via the UART bus. The vehicle processor then transmits the perception result to the vehicle's infotainment system or other vehicle ECUs via the CAN bus.

[0126] exist Figure 5 In the process, the serializer is connected to the vehicle's infotainment system or ECU via the LVDS bus, and the ignition switch is connected to the vehicle's processor via the ACC bus controlled by the key.

[0127] Figure 5 In the process, the power supply in the OMS vision perception processing board is provided by the automotive power supply. The power supply powers the vision perception processing chip. At the same time, the power supply is converted by the power converter to power the various modules (image sensor and image acquisition device) in the OMS system sensing module.

[0128] The aforementioned in-vehicle OMS system consists of two parts: the OMS vision perception processing board and the OMS system sensing module. The OMS vision perception processing board includes a vision perception processing chip, an in-vehicle processor MCU, a serializer, memory, and a power supply. The OMS system sensing module includes an image sensor and an image acquisition device. The OMS system sensing module transmits the MIPI signal, RST control signal, and MCLK clock signal of the transmitted image to the OMS vision perception processing board via an FPC cable. After receiving the image data from the OMS system sensing module, the OMS vision perception processing board processes the image using a vision perception algorithm on the vision perception processing chip to obtain the perception result. This result is then transmitted to the in-vehicle processor via the UART bus. The in-vehicle processor then transmits the perception result to the vehicle's infotainment system or other in-vehicle ECUs via the CAN bus.

[0129] Meanwhile, in this embodiment, the visual perception processing chip transmits the image to a serializer via MIPI, and the serializer transmits the image to the vehicle's infotainment system or other vehicle ECUs via LVDS. The OMS visual perception processing board is directly powered by automotive voltage levels, and a power converter converts the electrical energy supplied by the automotive voltage levels to power the OMS system's sensing modules.

[0130] The following is a schematic illustration of the workflow of the hardware system of the vehicle-mounted OMS system. The workflow includes:

[0131] 1. The image acquisition device is powered on and begins self-testing. At the same time, the image sensor is initialized.

[0132] 2. After the image sensor completes initialization, control the image acquisition device to acquire images of a predetermined space area inside the vehicle;

[0133] 3. The MIPI signal, RST control signal and MCLK clock signal of the transmitted image are transmitted to the OMS vision perception processing board through the FPC cable. The vision perception processing chip performs ISP (image signal processing) debugging on the image.

[0134] 4. Process the debugged image using a visual perception algorithm to obtain the perception result;

[0135] 5. The visual perception processing chip transmits the perception results to the vehicle processor via the UART bus. At the same time, it outputs the image to the serializer via MIPI, and then the serializer outputs the image to the engine or other vehicle ECUs via LVDS.

[0136] 6. After receiving the sensing results, the on-board processor transmits the sensing results to the vehicle's infotainment system or other on-board ECUs via the CAN bus.

[0137] The following is an illustrative explanation of the face registration and face recognition process in the OMS system.

[0138] Before using the facial recognition function of the in-vehicle OMS system, users need to register their faces. Users can upload their facial images to the system by following the system prompts or by actively uploading or inputting their own information. The system will then recognize the facial images, obtain facial features (including facial features, features of various points on the face, etc.), and save the facial features to the memory.

[0139] After completing face registration, users can use the system's face recognition function. For users who need face recognition, the face image collected by the OMS system sensor module can be transmitted to the OMS vision perception processing board. The vision perception processing chip extracts face features and compares the extracted features with the face features stored in the memory. The comparison result, i.e., the perception result, is sent to the vehicle processor. Then, the vehicle processor can send the perception result to the vehicle's infotainment system or other vehicle ECUs via the CAN protocol.

[0140] The following is an illustrative explanation of the fault diagnosis operation of the OMS system.

[0141] 1. The vision perception processing chip periodically monitors the working status of each hardware component in the OMS system. If the working status of any hardware component indicates a fault, a fault alarm is sent to the vehicle processor via UART.

[0142] 2. After receiving a fault alarm, the on-board processor updates and stores the visual perception chip fault information in the system fault register.

[0143] Meanwhile, the vehicle processor periodically monitors the power supply's operating status. If the power supply's operating status indicates a power supply failure, the power supply fault register in the vehicle processor is updated and the power supply fault information is stored.

[0144] 3. The on-board processor sends the visual perception chip fault information and power supply fault information to the vehicle's infotainment system or other on-board ECUs via the CAN bus according to the fault diagnosis protocol.

[0145] The following is an illustrative explanation of OTA upgrades for the OMS system.

[0146] 1. The vehicle's infotainment system or other vehicle ECUs initiate an OTA upgrade request and send the OTA upgrade package to the vehicle's processor via the CAN bus;

[0147] 2. The vehicle processor parses the OTA upgrade package and stores part of the OTA firmware in the internal storage of the vehicle processor;

[0148] 3. The onboard processor sends the OTA firmware of the vision perception processing chip to the vision perception processing chip;

[0149] 4. The visual perception processing chip saves its OTA firmware to the memory;

[0150] 5. The visual perception processing chip performs an OTA upgrade. After the upgrade is completed, the upgrade result (i.e., the first upgrade result) is sent to the vehicle processor via the UART bus.

[0151] 6. The vehicle processor performs an OTA upgrade. After the upgrade is completed, the upgrade result (i.e., the second upgrade result) is sent to the vehicle's infotainment system or other vehicle ECUs.

[0152] It should be noted that the system integrated in this application can also be a DMS and OMS integrated system. In this case, the DMS system and the OMS system share a camera, and the system sensing module and visual perception processing board are integrated in the camera.

[0153] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0154] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0159] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An in-vehicle cabin sensing device, characterized by, The vehicle-mounted cabin perception device is integrated with an image acquisition device, an image sensor, a visual perception processing chip, and a vehicle-mounted processor, wherein The image acquisition device is configured to acquire an image to be processed. An input end of the image sensor is connected to an output end of the image acquisition device, configured to receive the image to be processed and convert the image to be processed into an image signal to be processed. An input end of the visual perception processing chip is connected to an output end of the image sensor, configured to receive the image signal to be processed, perform image signal processing debugging on the image to be processed, and perform visual perception processing on the image signal to be processed by a neural network processing unit of the visual perception processing chip to obtain a perception result. An input end of the vehicle-mounted processor is connected to an output end of the visual perception processing chip, configured to receive the perception result, wherein the vehicle-mounted processor comprises a system fault register configured to monitor a working state of the visual perception processing chip, and update and store visual perception processing chip fault information if the visual perception processing chip fails; and a power supply fault register configured to monitor a working state of a power supply, and update and store power supply fault information if the power supply fails.

2. The in-vehicle cabin sensing device according to claim 1, characterized by, The visual perception processing on the image signal to be processed to obtain the perception result comprises: reading pre-stored object feature points; extracting image feature points of the image signal to be processed; comparing the object feature points with the image feature points to obtain the perception result.

3. The in-vehicle cabin sensing device according to claim 2, characterized by, The vehicle-mounted cabin perception device is further integrated with a memory, wherein the memory is connected to the visual perception processing chip and configured to store the object feature points.

4. The in-vehicle cabin sensing device according to any one of claims 1 to 3, characterized by, The vehicle-mounted cabin perception device is further integrated with one or more vehicle-mounted modules, wherein an input end of the one or more vehicle-mounted modules is connected to an output end of the vehicle-mounted processor and configured to receive the perception result.

5. The in-vehicle cabin sensing device according to claim 4, characterized by, The vehicle-mounted cabin perception device is further integrated with a serializer, wherein an input end of the serializer is connected to an output end of the visual perception processing chip, and an output end of the serializer is connected to an input end of the one or more vehicle-mounted modules, configured to receive the image signal to be processed and output the image signal to be processed to the one or more vehicle-mounted modules.

6. The in-vehicle cabin sensing device according to claim 4, characterized by, The vehicle-mounted cabin perception device is further integrated with a power supply, wherein an output end of the power supply is connected to an input end of the visual perception processing chip and configured to supply power to the visual perception processing chip.

7. The in-vehicle cabin sensing device according to claim 6, characterized by, The vehicle-mounted cabin perception device is further integrated with a power converter, wherein an input end of the power converter is connected to an output end of the power supply, and an output end of the power converter is connected to input ends of the image acquisition device and the image sensor, configured to convert the power supply and supply power to the image acquisition device and the image sensor.

8. The in-vehicle cabin sensing device according to claim 1, characterized by, The vehicle-mounted processor sends the visual perception processing chip fault information and the power supply fault information to the one or more vehicle-mounted modules according to a fault diagnosis protocol. 9.A method of controlling an in-vehicle cabin sensing device, characterized by, The control method is applied to the vehicle-mounted cabin perception device of any one of claims 1 to 8, and the control method comprises: acquiring an image to be processed. Integrate the to-be-processed image into a to-be-processed image signal; Perform image signal processing debugging on the to-be-processed image, and perform visual perception processing on the to-be-processed image signal in a neural network processing unit of the visual perception processing chip to obtain a perception result; Transmit the perception result to a vehicle-mounted processor, wherein a system fault register in the vehicle-mounted processor monitors a working state of the visual perception processing chip, and if the visual perception processing chip fails, updates and stores visual perception processing chip fault information; a power supply fault register in the vehicle-mounted processor monitors a working state of a power supply, and if the power supply fails, updates and stores power supply fault information. 10.The control method of a vehicle cabin sensing device according to claim 9, characterized by, Further comprising: Receiving a service upgrade request initiated by one or more vehicle-mounted modules; In response to the service upgrade request, sending a service upgrade package to the vehicle-mounted processor; The vehicle-mounted processor parses the service upgrade package and stores part of the service upgrade firmware; The vehicle-mounted processor sends the service upgrade firmware of the visual perception processing chip to the visual perception processing chip; The visual perception processing chip saves the service upgrade firmware into a memory; The visual perception processing chip performs service upgrade, and after the upgrade is completed, sends a first upgrade result to the vehicle-mounted processor; The vehicle-mounted processor performs service upgrade, and after the upgrade is completed, sends a second upgrade result to the one or more vehicle-mounted modules.

Images