A method, system and related products for integrating driver and passenger monitoring

Abstract

The present invention discloses a driver and passenger monitoring fusion method, system and related products, which relates to the technical field of intelligent driving monitoring systems, and solves the problems that the conventional DMS / OMS system fusion scheme leads to an increase in the control signal lines between the CIS sensor and the vehicle system, an increase in the computing power requirement for the vehicle system, and an increase in the integration difficulty; the key technical solution is: using an image sensor to realize the monitoring fusion of the driver area image and the in-cabin panoramic image, reducing the connection of the control line with the vehicle system, reducing the computing power requirement for the vehicle system, and reducing the integration difficulty; switching the pixel readout method within one frame interval to obtain a low-frame-rate in-cabin panoramic image and a high-frame-rate driver area image, meeting the passenger state monitoring requirement and the high-precision driver behavior monitoring requirement; switching the pixel readout method during the vertical blanking interval to reduce the occurrence of waste frames / bad frames and ensure the smoothness and stability of the image screen.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving monitoring systems, and more specifically, to a driver and passenger monitoring fusion method, system and related products. Background Art

[0002] With the continuous development of autonomous driving technology and the acceleration of commercialization, more and more new cars on the market are equipped with DMS (Driver Monitoring System) and OMS (Occupancy Monitoring System). DMS usually uses a small-format, high-frame-rate camera installed in front of the steering wheel or on the left front to monitor whether the driver's behavior is normal. It warns the driver when there is a violation and improves driving behavior, thereby improving driving safety. Correspondingly, OMS usually uses a large-format, low-frame-rate camera installed above the center console to monitor whether the passenger's behavior and condition are abnormal.

[0003] like Figure 1 As shown in the figure, the common DMS is to install a high-frame rate CIS (Contact Image Sensor) in the front row of the car to collect the driver's facial movement information at high speed, monitor the driver's behavior through the vehicle-mounted system (MCU), and alarm the abnormal driver behavior through the vehicle-mounted alarm device to achieve DMS coverage. Figure 2 As shown in the figure, the common OMS is to install a low-frame rate CIS in the rear seat of the car at the same time, collect the behavior and action information of the rear passengers at a low speed, transmit it to the car screen through the car system, and monitor the passenger behavior through the car screen to achieve OMS coverage. In the conventional DMS and OMS system integration solution, two CIS need to be connected to the car system, resulting in an increase in the control signal lines between the CIS and the car system, increasing the computing power requirements of the car system, and the difficulty of integration.

[0004] Therefore, the present application provides a driver and passenger monitoring fusion method, system and related products to solve the above problems. Summary of the invention

[0005] The purpose of this application is to provide a driver and passenger monitoring fusion method, system and related products, which solve the problems caused by the conventional DMS and OMS system fusion solutions, such as the increase in the control signal lines between the CIS sensor and the vehicle system, the increase in the computing power requirements of the vehicle system, and the increase in the integration difficulty. This solution only uses one image sensor to achieve full coverage of the in-vehicle scene. By controlling the exposure timing and readout method of the image sensor, without increasing the clock frequency, it achieves the fusion of DMS / OMS high and low frame rate synchronous image transmission application scenarios, reduces the connection of control signal lines, reduces the computing power requirements for the vehicle system, and reduces the software and hardware integration difficulty and cost.

[0006] This application first provides a driver and passenger monitoring fusion method, which is executed based on an image sensor system and includes: obtaining driver area coordinate information; determining the pixel readout method in the vertical blanking interval according to the driver area coordinate information; initiating a first exposure on the pixel matrix and obtaining a panoramic in-vehicle image through FULL readout; outputting the panoramic in-vehicle image through the main path data interface for monitoring the passenger state; obtaining a first driver area image by cropping and adding the panoramic in-vehicle image, and outputting the first driver area image through the branch path data interface after preprocessing for monitoring the driver's behavior. The cropping and adding process includes: extracting the first driver area image from the panoramic in-vehicle image according to the driver area coordinate information and the pixel readout method in the vertical blanking interval; when the frame enters the vertical blanking interval, initiating a second exposure on the pixel matrix and obtaining a second driver area image through the pixel readout method in the vertical blanking interval, and outputting the second driver area image through the branch path data interface after preprocessing.

[0007] In a possible implementation, determining the pixel readout method in the vertical blanking interval according to the driver area coordinate information includes: calculating the driver area size according to the start coordinate signal and end coordinate information in the driver area coordinate information; comparing the driver area size with the transmittable size in the vertical blanking interval of the FULL readout of the image sensor, and determining the pixel readout method in the vertical blanking interval according to the comparison result.

[0008] In a possible implementation, the size of the driver area is compared with the transmittable size of the field blanking interval read out by the image sensor FULL, and the readout method of the pixels in the field blanking interval is determined according to the comparison result, including: if the size of the driver area does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout method is switched from Full readout to ROI readout; if the size of the driver area exceeds the transmittable size of the field blanking interval, the V2H2 addition process is performed on the size of the driver area. If the size of the driver area after the V2H2 addition process does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout method is switched from Full readout to V2H2 addition readout; if the size of the driver area after the V2H2 addition process still exceeds the transmittable size of the field blanking interval, the V4H4 addition process is performed on the size of the driver area. If the size of the driver area after the V4H4 addition process does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout method is switched from Full readout to V4H4 addition readout.

[0009] In a possible implementation, the preprocessing includes: filtering and noise reduction and contour information extraction processing.

[0010] In a possible implementation, the contour information extraction processing is performed using the local binary method

[0011] The present application also provides a driver and passenger monitoring fusion system, including: a vehicle-mounted system, an image sensor system, a vehicle-mounted display screen, and a vehicle-mounted alarm device; the vehicle-mounted system is used to obtain the driver's face image, perform face recognition based on the driver's face image, and when the face recognition fails, re-send an instruction for obtaining the driver's face image to the image sensor system. When the face recognition passes, obtain the driver area coordinate information; the image sensor system is used to execute a driver and passenger monitoring fusion method as described above to obtain a panoramic image of the cabin, a first driver area image, and a second driver area image; the vehicle-mounted system is used to receive the panoramic image of the cabin output by the main path data interface, transmit the panoramic image of the cabin to the vehicle-mounted display screen, and is used to receive the first driver area image and the second driver area image output by the branch path data interface, and monitor the driver's behavior in real time according to the first driver area image and the second driver area image. When the driver's behavior is abnormal, an alarm message is sent to the vehicle-mounted alarm device; the vehicle-mounted display screen is used to monitor the passenger status in real time according to the panoramic image of the cabin; the vehicle-mounted alarm device is used to emit a sound and / or light alarm in response to the alarm message.

[0012] In a possible implementation, the driver behavior is monitored in real time according to the first driver area image and the second driver area image; including: the first driver area image and the second driver area image are collectively referred to as the driver area image, the contour information of the driver area image is matched with the driver face feature data pre-stored locally, if the match fails, a warning is issued and new contour information is received for matching, if the match is successful, the current contour information is locked; the feature parts in the contour information are calibrated, and the position information and the opening amplitude information of the feature parts are captured; the position information and the opening amplitude information of the feature parts are input into the fatigue state evaluation model to predict whether the driver behavior is abnormal.

[0013] The present application also provides a computer device including: a memory, one or more processors, and the processors are used to execute the computer programs or instructions stored in the memory so that the computer device executes a driver and passenger monitoring fusion method as described above.

[0014] The present application also provides a computer storage medium including computer programs or instructions, and when the computer programs or instructions are run on a computer device, the computer device is enabled to execute a driver and passenger monitoring fusion method as described above.

[0015] The present application also provides a computer program product, and when the computer program product is run on a computer device, the computer device is enabled to execute a driver and passenger monitoring fusion method as described above.

[0016] Compared with the prior art, the present application has the following beneficial effects: the present application uses one image sensor to realize the monitoring fusion of the driver area image and the in-cabin panoramic image, reduces the connection of the control line with the vehicle-mounted system, reduces the computing power requirement for the vehicle-mounted system, and reduces the integration difficulty; the pixel readout method is switched within one frame interval to obtain a low-frame-rate in-cabin panoramic image and a high-frame-rate driver area image, meeting the passenger state monitoring requirement and the high-precision driver behavior monitoring requirement; the pixel readout method is switched during the vertical blanking interval, reducing the occurrence of waste frames / bad frames and ensuring the smoothness and stability of the image screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, form a part of the present application, and do not limit the embodiments of the present invention. In the drawings:

[0018] Figure 1 is a common DMS structure diagram;

[0019] Figure 2 is a common OMS structure diagram;

[0020] Figure 3Schematic flowchart of the driver and passenger monitoring integration method provided by the embodiments of the present application;

[0021] Figure 4 Schematic diagram of clipping addition and field blanking readout in the case of ROI readout;

[0022] Figure 5 Schematic diagram of clipping addition and field blanking readout in the case of V2H2 readout;

[0023] Figure 6 Schematic diagram of clipping addition and field blanking readout in the case of V4H4 readout;

[0024] Figure 7 Schematic diagram of the main path data interface and branch path data interface images obtained by switching exposure readout;

[0025] Figure 8 Schematic diagram of dynamic switching exposure readout of a traditional image sensor;

[0026] Figure 9 Schematic diagram of dynamic switching exposure readout of the image sensor of the present application;

[0027] Figure 10 Schematic diagram of the driver and passenger monitoring integration system provided by the embodiments of the present application;

[0028] Figure 11 Workflow diagram of the driver and passenger monitoring integration system;

[0029] Figure 12 Schematic diagram of the structure of the computer device provided by the embodiments of the present application. Detailed implementation manners

[0030] Hereinafter, the term "comprise" or "may comprise" that may be used in various embodiments of the present application indicates the presence of the claimed functions, operations or elements, and does not limit the addition of one or more functions, operations or elements. Further, as used in various embodiments of the present application, the terms "comprise", "have" and their cognates are only intended to mean the presence of specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be construed as precluding the existence or possibility of addition of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items first.

[0031] In various embodiments of the present application, expressions (such as "first", "second", etc.) used may modify various constituent elements in the various embodiments, but do not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only for the purpose of distinguishing one element from other elements. For example, the first user device and the second user device indicate different user devices, although both are user devices. For example, without departing from the scope of the various embodiments of the present application, the first element may be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0032] The terms used in the various embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the various embodiments of the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in a general use dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the present application.

[0033] To make the purpose, technical solution and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with embodiments and drawings. The illustrative embodiments of the present application and their descriptions are only for explaining the present application and do not serve as a limitation to the present application.

[0034] Please refer to Figure 3 as shown Figure 3 is a schematic flowchart of a driver and passenger monitoring fusion method provided by an embodiment of the present application. The driver and passenger monitoring fusion method is executed based on an image sensor system and includes: obtaining driver area coordinate information; determining a pixel readout method for the vertical blanking interval according to the driver area coordinate information; initiating a first exposure to the pixel matrix and obtaining a panoramic image of the cabin through FULL readout; outputting the panoramic image of the cabin through the main path data interface for monitoring the passenger state; obtaining a first driver area image by performing cropping and addition processing on the panoramic image of the cabin, and outputting the first driver area image through the branch path data interface after preprocessing for monitoring the driver's behavior. The cropping and addition processing includes: extracting the first driver area image from the panoramic image of the cabin according to the driver area coordinate information and the pixel readout method for the vertical blanking interval; when a frame enters the vertical blanking interval, initiating a second exposure to the pixel matrix and obtaining a second driver area image through the pixel readout method for the vertical blanking interval, and outputting the second driver area image through the branch path data interface after preprocessing.

[0035] Specifically, the image sensor system obtains driver area coordinate information from the vehicle control system, and determines the switching mode of the pixel readout method during the frame field blanking interval based on this information. The exposure readout control module initiates the first exposure readout control for the pixel matrix, and obtains a panoramic image of the cabin through the FULL (full size) readout method. The panoramic image of the cabin is processed through two data paths respectively. On the main path, the panoramic image of the cabin is directly output through the main path data interface without any other special processing. The panoramic image of the cabin on the branch path is transmitted to the cropping and addition module. The scaling factor, cropping position and size are determined by the previously obtained driver area coordinate information and the pixel readout method during the field blanking interval, and the driver area image is extracted from the panoramic image of the cabin to obtain the first driver area image. Then, it is processed by a preprocessing module for filtering and noise reduction, contour information extraction, etc., reducing the computational workload for subsequent driver behavior monitoring by the vehicle system, and finally output through the branch path data interface. After obtaining the panoramic image of the cabin and the first driver area image, when the current frame enters the field blanking interval, the exposure readout control module initiates the second exposure readout control for the current frame for the pixel matrix unit, and the readout method is dynamically switched from FULL readout to the pixel readout method during the previously obtained field blanking interval, realizing fast exposure readout only for the driver area to obtain the second driver area image. The second driver area image is preprocessed through the branch path. Since the image obtained by the second exposure readout only contains the driver area, the cropping and addition module does not process the second driver area image at this time, but directly performs processing such as filtering and noise reduction, and contour information extraction through the pre-module, extracts key information, and finally outputs through the branch path data interface. The low-frame-rate panoramic image of the cabin received by the main path data interface can be used for subsequent passenger status monitoring on the vehicle display screen. The contour information of the high-frame-rate driver area images (the first driver area image and the second driver area image) output by the branch path data interface can be used for driver behavior monitoring by the vehicle system.

[0036] The improvements of this application are as follows: First, the number of image sensors is reduced from two to one, which reduces the requirements for the number of interfaces with the vehicle system, simplifies the wiring difficulty with the vehicle system, reduces the requirements for the computing power of the vehicle system, and at the same time realizes stable integration and coverage of DMS and OMS; Second, two exposures are set within one frame to obtain a panoramic image of the cabin and two driver area images, performing high-frequency acquisition on the driver area and low-frequency acquisition on the panoramic image of the cabin, ensuring the ability to monitor the passenger status while accurately tracking the driver's behavior and avoiding dangerous driving; Third, the readout method is switched during the field blanking interval, which can reduce the situation of discarded frames / bad frames existing in traditional exposure readout switching, ensuring the smoothness and stability of the image screen.

[0037] It should be noted that vertical blanking (VBlank) refers to the idle period before the start of the next frame data transfer after the end of the transfer of one frame of data, during which the functional module is inactive. The size that can be transmitted during the vertical blanking interval refers to the calculation based on the frame rate of the current reading method. If the frame rate is fixed, that is, the time of one frame is fixed, and the frame size and transmission rate of one frame of image data are fixed, then the time required for transmitting one frame of image data can be calculated. Subtracting the time required for transmitting one frame of image data from the time of one frame can obtain the duration of the vertical blanking interval, and the amount of data that can be transmitted within this time is the size that can be transmitted during the vertical blanking interval.

[0038] Assume that the driver area is fixed and the sensor model is fixed. Then, the reading method of the pixels in the vertical blanking interval can be directly preset according to the fixed coordinate information of the driver area and the fixed sensor model to ensure that an image of the driver area can be obtained after switching the reading method in the vertical blanking interval. However, the actual situation is more complex. The driver areas of different vehicle models may be different, the image sizes read by the FULL of different models of image sensors are different, and the sizes that can be transmitted during the vertical blanking interval are also different. Therefore, it is difficult to meet the usage requirements in complex scenarios by using a single preset reading method for the pixels in the vertical blanking interval. For this reason, this application proposes a method for determining the reading method of the pixels in the vertical blanking interval to meet the usage requirements in complex scenarios.

[0039] In a possible implementation manner, determining the reading method of the pixels in the vertical blanking interval according to the coordinate information of the driver area includes: calculating the size of the driver area according to the start coordinate signal and the end coordinate information in the coordinate information of the driver area; comparing the size of the driver area with the size that can be transmitted during the vertical blanking interval read by the FULL of the image sensor, and determining the reading method of the pixels in the vertical blanking interval according to the comparison result.

[0040] Further, compare the driver area size with the transmissible size of the field blanking interval read out by the image sensor in FULL mode, and determine the readout method of the pixels in the field blanking interval according to the comparison result, including: if the driver area size does not exceed the transmissible size of the field blanking interval, send the driver area coordinate information to the pixel matrix at the start of the field blanking interval of the current frame, and switch the readout method from Full readout to ROI readout; if the driver area size exceeds the transmissible size of the field blanking interval, perform V2H2 addition processing on the driver area size. If the driver area size after V2H2 addition processing does not exceed the transmissible size of the field blanking interval, send the driver area coordinate information to the pixel matrix at the start of the field blanking interval of the current frame, and switch the readout method from Full readout to V2H2 addition readout; if the driver area size after V2H2 addition processing still exceeds the transmissible size of the field blanking interval, perform V4H4 addition processing on the driver area size. If the driver area size after V4H4 addition processing does not exceed the transmissible size of the field blanking interval, send the driver area coordinate information to the pixel matrix at the start of the field blanking interval of the current frame, and switch the readout method from Full readout to V4H4 addition readout.

[0041] Specifically, compare the driver area size with the transmissible size of the field blanking interval read out by the image sensor in FULL mode, and determine the readout method of the pixels in the field blanking interval according to the comparison result. When the driver area size is less than or equal to the transmissible size of the field blanking interval read out by the image sensor in FULL mode, it means that the field blanking interval read out in FULL mode can transmit the image of the driver area. At this time, only need to change the pixel readout method to ROI (Region of Interest) readout to read out the image of the driver area. When the driver area size is greater than the transmissible size of the field blanking interval read out by the image sensor in FULL mode, it means that the field blanking interval read out in FULL mode cannot transmit the image of the driver area. At this time, it is necessary to change the pixel readout method, such as using V2H2 addition (merging different colors of 2 rows and 2 columns) readout, V4H4 addition (merging different colors of 4 rows and 4 columns) readout, etc., to reduce the number of read pixels and output the driver area image at a lower resolution.

[0042] It can be understood that switching from Full readout to ROI readout, V2H2 addition readout or V4H4 addition readout can perform secondary exposure readout of the driver's seat area separately in an extremely short time. It is equivalent to being able to complete two readouts of the driver's seat area image while reading out the panoramic image of the passenger compartment once, realizing high-frame-rate readout of the driver area image, meeting the high-frame-rate motion detection requirements of DMS, and then completing the integration of DOMS (Driver and Occupancy Monitoring System).

[0043] The following combinesFigures 4-6 The working process of the image sensor system is described in detail for the three possible readout methods (ROI readout, V2H2 readout, V4H4 readout).

[0044] When the image sensor system starts up, after receiving the pre-calibrated driver area coordinate information from the vehicle system, it calculates the driver area size through the start and end coordinate information in the driver area coordinate information, compares the driver area size with the transmissible size of the vertical blanking interval of the current image sensor readout method (Full), and presets the pixel readout method for the vertical blanking interval according to the comparison result; at the same time, it sends this driver area coordinate information and the preset pixel readout method for the vertical blanking interval to the cropping and addition module. The driver area coordinate information can ensure that the driver area image is accurately cropped from the in-cabin panoramic image, and the pixel readout method for the vertical blanking interval can ensure that a suitable addition operation is matched so that the vertical blanking interval can completely transmit the driver area image, ensuring the continuity of the image.

[0045] The three readout methods for the vertical blanking interval are as follows:

[0046] Case 1: As Figure 4 shown, Figure 4 is a schematic diagram of cropping, addition, and vertical blanking readout in the case of ROI readout. As Figure 4 shown in the lower part, if the current driver area size does not exceed the transmissible size of the vertical blanking interval, then the exposure readout control module will send the driver area coordinate information to the pixel matrix at the start of the vertical blanking of the current frame, and the readout method will switch from Full readout to ROI readout to ensure that the driver area image is obtained by secondary exposure readout in the vertical blanking interval. As Figure 4 shown in the upper part, the exposure readout control module will also send the driver area coordinate information and the pixel readout method for the vertical blanking interval to the cropping and addition module to determine the cropping start position and addition information (ROI readout does not perform addition processing). The cropping and addition function will act on the in-cabin panoramic image read out by one-time exposure FULL to crop and add to obtain the driver area image.

[0047] Case 2: As Figure 5 shown, Figure 5 is a schematic diagram of cropping, addition, and vertical blanking readout in the case of V2H2 readout. As Figure 5As shown in the lower part, if the size of the current driver area exceeds the transmissible size of the vertical blanking interval, the size of the driver area after V2H2 addition is compared with the transmissible size of the vertical blanking interval. If the size of the driver area after V2H2 addition does not exceed the transmissible size of the vertical blanking interval, the readout control module will send the driver area coordinate information to the pixel matrix at the start of the vertical blanking of the current frame, and the readout method will be switched from Full readout to V2H2 addition readout to ensure that the driver area image is obtained by double exposure readout in the vertical blanking interval. As Figure 5 As shown in the upper part, the exposure readout control module will also send the driver area coordinate information and the pixel readout method in the vertical blanking interval to the cropping and addition module to determine the cropping start position and the addition information (V2H2 readout is processed by V2H2 addition). The cropping and addition function will act on the full cabin panoramic image read out by single exposure FULL to crop and add to obtain the driver area image.

[0048] Case 3: As Figure 6 shown Figure 6 is a schematic diagram of cropping and addition and vertical blanking readout in the case of V4H4 readout. As Figure 6 As shown in the lower part, if the size of the current driver area exceeds the transmissible size of the vertical blanking interval, the size of the driver area after V2H2 addition is compared with the transmissible size of the vertical blanking interval. If the size of the driver area after V2H2 addition still exceeds the transmissible size of the vertical blanking interval, the size of the driver area after V4H4 addition is compared with the transmissible size of the vertical blanking interval. If the size of the driver area after V4H4 addition does not exceed the transmissible size of the vertical blanking interval, the readout control module will send the driver area coordinate information to the pixel matrix at the start of the vertical blanking of the current frame, and the readout method will be switched from Full readout to V4H4 addition readout to ensure that the driver area image is obtained by double exposure readout in the vertical blanking interval. As Figure 6 As shown in the upper part, the exposure readout control module will also send the driver area coordinate information and the pixel readout method in the vertical blanking interval to the cropping and addition module to determine the cropping start position and the addition information (V4H4 readout is processed by V4H4 addition). The cropping and addition function will act on the full cabin panoramic image read out by single exposure FULL to crop and add to obtain the driver area image.

[0049] Please refer to Figure 7 shown Figure 7 is a schematic diagram of the main path data interface and the branch path data interface images obtained by switching exposure readout. In the figure, XVS is the vertical synchronization signal, N Frame is the Nth frame, and N+1 Frame is the (N + 1)th frame. It can realize a low frame rate image sensor to transmit two-way high and low frame rate image data simultaneously.

[0050] It should be noted that, in addition to ROI readout, V2H2 readout, and V4H4 readout, the pixel readout method in the vertical blanking interval can also be other common pixel merging modes, such as NxN Binning: where "N" can be any positive integer, indicating that N pixels are merged into one pixel in the horizontal and vertical directions. Another example is discarding some pixels, irregularly merging pixels (such as 1x2, 2x3, etc.), combining irregularly merging pixels with discarding some pixels, etc. ROI, V2H2, and V4H4 do not constitute a limitation on the pixel readout method in the vertical blanking interval. Any pixel readout method in the vertical blanking interval that makes the size of the driver area not exceed the transmissible size of the vertical blanking interval can be used as the pixel readout method in the vertical blanking interval of this application.

[0051] It can be understood that for the driver and passenger monitoring fusion method proposed in this application, the image sensor system needs to dynamically switch the image exposure readout mode during the working mode to obtain three image data within one frame time. The main path outputs a low-frame-rate in-cabin panoramic image for OMS monitoring, and the branch path outputs two high-frame-rate driver area images for DMS monitoring, realizing the simultaneous output of high- and low-frame-rate images.

[0052] As Figure 8 shown, Figure 8 is a schematic diagram of the dynamic switching of exposure readout of a traditional image sensor. When the traditional image sensor dynamically switches the exposure readout, since the exposure of the pixel data read in a frame starts in the previous frame, the dynamic switching of the exposure readout mode will cause the pixel exposure and pixel readout method of the front and rear frames to not match, resulting in an abnormal image frame after the mode switch, making the output image of the image sensor discontinuous, and there are bad frames / waste frames in the middle, affecting the later driver behavior monitoring and causing false alarms in driver behavior monitoring. Please refer to Figure 9 shown, Figure 9 is a schematic diagram of the dynamic switching of exposure readout of the image sensor of this application. This application proposes to dynamically switch the exposure readout in the vertical blanking interval, effectively avoiding the appearance of bad frames / waste frames between frames, ensuring the continuity of the output image, and improving the accuracy of in-cabin monitoring.

[0053] In a possible implementation manner, the preprocessing includes: filtering and noise reduction and contour information extraction processing. Specifically, in the image sensor system of this application, two data paths with different frame rates, the main path and the branch path, are obtained, which respectively meet the monitoring requirements of OMS and DMS. In order to further reduce the data reception difficulty and operation difficulty of the backend vehicle-mounted system, this application also proposes to pre-set a preprocessing module inside the image sensor system to perform preprocessing on the driver area image, extract key information after filtering and noise reduction, and simplify the transmitted data to facilitate the backend vehicle-mounted system to perform driver behavior monitoring.

[0054] Further, the contour information extraction process is carried out using the local binary method. Specifically, the driver behavior monitoring mainly needs to collect features such as the driver's eyes, nose, mouth, and facial contour. In this application, LBP (local binary pattern) can be used to extract the facial features of the human face. Utilizing the characteristic that the LBP operator has feature invariance to the overall illumination, it can maintain the contrast when the light in the cockpit changes. The LBP feature can well handle the overall illumination interference and local weak interference, and accurately extract the facial feature information of the human face. Specifically, a 3x3 pixel matrix is defined on the branch path. Taking the gray value of the central pixel of this matrix as the threshold, the gray values of the surrounding 8 pixels are compared with it. If the gray value of the surrounding pixel is greater than the gray value of the central pixel, the pixel gray value is marked as 1, otherwise it is marked as 0. After all 8 surrounding pixel points in the 3x3 pixel matrix are compared and marked, an 8-bit binary number can be obtained. This value is the LBP value of the central pixel of the 3x3 pixel matrix, and this LBP value can reflect the texture information within the range of this matrix. By sequentially scanning the driver area image using the LBP operator, the feature information of the driver area image can be obtained.

[0055] Please refer to Figure 10 as shown in Figure 10 FIG. [FIGURE NUMBER] is a schematic diagram of the driver and passenger monitoring fusion system provided by an embodiment of the present application. The system includes: a vehicle-mounted system, an image sensor system, a vehicle-mounted display screen, and a vehicle-mounted alarm device; the vehicle-mounted system is used to obtain the driver's facial image, perform face recognition based on the driver's facial image, and when the face recognition fails, re-send an instruction for obtaining the driver's facial image to the image sensor system, and when the face recognition passes, obtain the driver area coordinate information; the image sensor system is used to execute a driver and passenger monitoring fusion method as described above to obtain a panoramic image of the cabin, a first driver area image, and a second driver area image; the vehicle-mounted system is used to receive the panoramic image of the cabin output by the main path data interface, transmit the panoramic image of the cabin to the vehicle-mounted display screen, and is used to receive the first driver area image and the second driver area image output by the branch path data interface, and monitor the driver's behavior in real time according to the first driver area image and the second driver area image. When the driver's behavior is abnormal, an alarm message is sent to the vehicle-mounted alarm device; the vehicle-mounted display screen is used to monitor the passenger status in real time according to the panoramic image of the cabin; the vehicle-mounted alarm device is used to emit a sound and / or light alarm in response to the alarm message.

[0056] Please refer to Figure 11 as shown in Figure 11 FIG. [FIGURE NUMBER] is a working flowchart of the driver and passenger monitoring fusion system. When the driver and passenger monitoring fusion system is working:

[0057] Note: In the above translation, [FIGURE NUMBER] should be replaced with the actual figure number in the original text. Since the figure number is not provided in the given text, it is left as a placeholder here.The vehicle-mounted system starts, performs driver matching recognition and stores the driver area coordinate information. After successful recognition, it controls the start of the image sensor system and sends the driver area coordinate information to the image sensor system.

[0058] The exposure readout control module of the image sensor system calculates the driver area size based on the driver area coordinate information and compares it with the transmissible size of the field blanking area under FULL readout to determine the pixel readout method in the field blanking interval of the image sensor, and sends the driver area coordinate information and the pixel readout method in the field blanking interval to the back-end cropping and addition module.

[0059] The pixel matrix is read out in FULL to obtain a panoramic image of the passenger compartment. On the one hand, the panoramic image of the passenger compartment enters the main path: the panoramic image of the passenger compartment is transmitted to the vehicle-mounted system through the main path data interface; the vehicle-mounted system sends the panoramic image of the passenger compartment to the vehicle-mounted screen for passenger status monitoring (OMS); after the transmission of one frame of the panoramic image of the passenger compartment in the main path is completed, the image sensor system enters the field blanking interval. On the other hand, the panoramic image of the passenger compartment enters the branch path: the panoramic image of the passenger compartment is transmitted to the cropping and addition module, and corresponding cropping and addition processing is performed according to the driver area coordinate information and the pixel readout method in the field blanking interval obtained from the exposure readout module to obtain the first driver area image; the first driver area image is transmitted to the preprocessing module for noise reduction filtering processing and contour information extraction processing to improve the picture clarity and reduce the data volume for the vehicle-mounted system to monitor driver behavior; the preprocessed first driver area image is transmitted to the vehicle-mounted system through the branch path data interface; the built-in MCU of the vehicle-mounted system monitors driver behavior and controls the vehicle-mounted alarm device to alarm when abnormal driver behavior is detected (DMS); after the transmission of one frame of the driver area image in the branch path is completed, the image sensor system enters the field blanking interval.

[0060] The image sensor system enters the field blanking interval: the exposure readout control module controls the pixel matrix to perform a second pixel exposure readout based on the previously obtained driver area coordinate information and the pixel readout method during the field blanking period to obtain the second driver area image; the second driver area image is transmitted to the preprocessing module for noise reduction filtering processing and contour information extraction processing to improve the picture clarity and reduce the data volume for the vehicle-mounted system to monitor driver behavior; the preprocessed second driver area image is transmitted to the vehicle-mounted system through the branch path data interface; the built-in MCU of the vehicle-mounted system monitors driver behavior and controls the vehicle-mounted alarm device to alarm when abnormal driver behavior is detected (DMS).

[0061] Specifically, the driver starts the vehicle, controls the image sensor system to collect the driver's facial images at a lower frame rate, and performs driver face recognition through the built-in MCU. If the recognition is abnormal (such as no portrait information is collected, the collected portrait information does not match the pre-recorded portrait information, etc.), an abnormal message is prompted on the in-vehicle screen and the image sensor system is re-controlled to collect the driver's facial images. After the recognition passes, the driver area coordinate information is extracted and sent to the image sensor system. The image sensor system defaults to FULL readout. Based on the received driver area coordinate information, the driver area size is calculated, that is, the size occupied by the driver area in the in-cabin panoramic image, and it is compared with the transmissible size of the field blanking interval size in the current image sensor readout mode to determine the pixel readout mode of the field blanking interval. The image sensor system starts to work, and the readout mode cycles through Full readout -> ROI / V2H2 / V4H4 readout, obtaining two images transmitted at high and low frame rates (the in-cabin panoramic image with a low frame rate on the main path, and the driver area image with a high frame rate on the branch path). The two paths of data are respectively transmitted to the back-end in-vehicle system through the two high and low frame rate data interfaces (the main path data interface and the branch path data interface) carried by the image sensor system. The in-vehicle system obtains the in-cabin panoramic image with a low frame rate and transmits it to the in-vehicle screen for real-time monitoring of the in-vehicle environment and the passenger status, realizing the OMS function. At the same time, the in-vehicle system obtains the driver area image with a high frame rate, transmits it to the built-in MCU for driver behavior monitoring, and when an abnormal driving position behavior is detected, a warning is given through the in-vehicle alarm device (such as a speaker), realizing the DMS function.

[0062] In a possible implementation manner, the driver behavior is monitored in real time according to the first driver area image and the second driver area image; it includes: the first driver area image and the second driver area image are collectively referred to as the driver area image. The contour information of the driver area image is matched with the driver face feature data pre-stored locally. If the match fails, a warning is issued and new contour information is received for matching. If the match is successful, the current contour information is locked; the feature parts in the contour information are calibrated, and the position information and the opening amplitude information of the feature parts are captured; the position information and the opening amplitude information of the feature parts are input into the fatigue state evaluation model to predict whether the driver behavior is abnormal.

[0063] Specifically, compare and match the driver area image with the pre-stored driver face feature data locally. If the match fails, issue a warning and extract a new input image for continuous matching. If the matching is completed, lock the current face features and continue with driver fatigue monitoring. Taking facial monitoring as an example, calibrate feature parts such as eyes and mouth, capture information such as the positions and opening amplitudes of the eyes and mouth at each moment. Through the statistics and analysis of this information, establish a fatigue state evaluation model in combination with multiple representative fatigue state evaluation indicators to monitor and judge the mental state of the driver at any time. At the same time, give an alarm reminder according to the monitoring results, and use the on-board alarm device to give a voice prompt to the driver to pay attention to driving safety, thus completing the real-time monitoring of DMS.

[0064] In summary, the present application uses one image sensor to realize the monitoring integration of the driver area image and the cabin panoramic image, reducing the connection of control lines to the vehicle-mounted system, reducing the computing power requirements for the vehicle-mounted system, and reducing the integration difficulty; switching the pixel readout method within one frame interval to obtain a low-frame-rate cabin panoramic image and a high-frame-rate driver area image, meeting the monitoring requirements for passenger status and high-precision driver behavior monitoring; switching the pixel readout method during the vertical blanking interval to reduce the occurrence of discarded frames / bad frames and ensure the smoothness and stability of the image.

[0065] Please refer to Figure 12 shown in Figure 12 which is a schematic structural diagram of the computer device provided by the embodiment of the present application. The computer device includes: a memory, one or more processors, and the processor is used to execute the computer programs or instructions stored in the memory so that the computer device executes a driver and passenger monitoring integration method as described above.

[0066] The computer device 1200 may vary greatly due to configuration or performance differences, and may include one or more central processing units (CPUs) 1222 (for example, one or more processors) and a memory 1232, and one or more storage media 1230 (for example, one or more mass storage devices) for storing application programs 1242 or data 1244. Among them, the memory 1232 and the storage media 1230 may be transient storage or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the computer device. Further, the central processor 1222 may be configured to communicate with the storage media 1230 and execute a series of instruction operations in the storage media 1230 on the computer device 1200.

[0067] The computer device 1200 may further include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input data interfaces 1258, and / or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so on.

[0068] In an embodiment of the present application, the central processing unit 1222 may be used to execute the above-mentioned method for fusing driver and passenger monitoring.

[0069] An embodiment of the present application further provides a computer storage medium, including a computer program or instruction. When the computer program or instruction runs on the computer device, the computer device is caused to execute a method for fusing driver and passenger monitoring as described above.

[0070] An embodiment of the present application further provides a computer program product. When the computer program product runs on the computer device, the computer device is caused to execute a method for fusing driver and passenger monitoring as described above.

[0071] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A driver and passenger monitoring fusion method, characterized in that: Image sensor-based system execution includes: Get the driver area coordinate information; Determine a pixel readout method in a field blanking interval according to the driver area coordinate information; Initiate the first exposure of the pixel matrix and obtain a panoramic view of the cabin through full-size readout; Outputting the cabin panoramic image through the main path data interface for monitoring passenger status; The first driver area image is obtained by cutting and adding the panoramic image in the cabin, and the first driver area image is output through the branch path data interface after preprocessing for monitoring the driver's behavior, wherein the cutting and adding processing includes: extracting the first driver area image from the panoramic image in the cabin according to the driver area coordinate information and the field blanking interval pixel reading method; When the frame enters the field blanking interval, a second exposure is initiated for the pixel matrix and a second driver area image is obtained by a field blanking interval pixel readout method, and the second driver area image is output through a branch path data interface after preprocessing.

2. A driver and passenger monitoring fusion method according to claim 1, characterized in that: Determining a pixel readout method in a field blanking interval according to the driver area coordinate information; comprising: Calculate the driver area size according to the starting coordinate signal and the ending coordinate information in the driver area coordinate information; The driver area size is compared with the transmittable size of the field blanking interval read out in full size by the image sensor, and the readout method of the field blanking interval pixels is determined according to the comparison result.

3. A driver and passenger monitoring fusion method according to claim 2, characterized in that: Comparing the driver area size with the field blanking interval transmittable size read out by the full size of the image sensor, and determining the readout method of the field blanking interval pixels according to the comparison result; comprising: When the driver area size does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode is switched from full-size readout to ROI readout; When the size of the driver area exceeds the transmittable size in the field blanking interval, V2H2 addition processing is performed on the driver area size. When the size of the driver area after V2H2 addition processing does not exceed the transmittable size in the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode is switched from full-size readout to V2H2 addition readout; When the size of the driver area after the V2H2 addition processing still exceeds the transmittable size of the field blanking interval, the V4H4 addition processing is performed on the driver area size; when the size of the driver area after the V4H4 addition processing does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode is switched from full-size readout to V4H4 addition readout; Among them, V2H2 addition refers to the merging of 2 rows and 2 columns of different colors, and V4H4 addition refers to the merging of 4 rows and 4 columns of different colors.

4. A driver and passenger monitoring fusion method according to claim 1, characterized in that: The preprocessing includes: filtering and noise reduction and contour information extraction processing.

5. A driver and passenger monitoring fusion method according to claim 4, characterized in that: The contour information extraction process is performed using a local binary method.

6. A driver and passenger monitoring fusion system, characterized in that: A driver and passenger monitoring fusion system comprising: In-vehicle systems, image sensor systems, in-vehicle displays and in-vehicle warning devices; The vehicle-mounted system is used to obtain a facial image of the driver, perform face recognition based on the facial image of the driver, resend an instruction for obtaining the facial image of the driver to the image sensor system when the face recognition fails, and obtain the driver area coordinate information when the face recognition passes; The image sensor system is used to execute a driver and passenger monitoring fusion method as described in any one of claims 1 to 5 to obtain a cabin panoramic image and a first driver area image and a second driver area image; The vehicle-mounted system is used to receive the panoramic image of the cabin output by the main path data interface, transmit the panoramic image of the cabin to the vehicle-mounted display screen, receive the first driver area image and the second driver area image output by the branch path data interface, monitor the driver's behavior in real time according to the first driver area image and the second driver area image, and send an alarm message to the vehicle-mounted alarm device when the driver's behavior is abnormal; The vehicle-mounted display screen is used to monitor the passenger status in real time based on the panoramic image in the cabin; The vehicle-mounted alarm device is used to emit sound and / or light alarm in response to the alarm information.

7. A driver and passenger monitoring fusion system according to claim 6, characterized in that: Real-time monitoring of driver behavior based on a first driver area image and a second driver area image; including: The first driver area image and the second driver area image are collectively referred to as the driver area image, and the contour information of the driver area image is matched with the driver's facial feature data pre-stored locally. If the match fails, a warning is issued and new contour information is received for matching. If the match succeeds, the current contour information is locked; The characteristic parts in the contour information are calibrated, and the position information and the opening amplitude information of the characteristic parts are captured; The position information and opening amplitude information of the characteristic parts are input into the fatigue state evaluation model to predict whether the driver's behavior is abnormal.

8. A computer device, characterized in that: include: A memory, one or more processors, wherein the processor is used to execute a computer program or instruction stored in the memory so that the computer device executes a driver and passenger monitoring fusion method as described in any one of claims 1-5.

9. A computer storage medium, characterized in that It comprises a computer program or an instruction, which, when executed on a computer device, enables the computer device to execute a driver and passenger monitoring fusion method as claimed in any one of claims 1 to 5.

10. A computer program product, characterized in that When the computer program product runs on a computer device, the computer device is enabled to execute a driver and passenger monitoring fusion method as described in any one of claims 1 to 5.

Images