Driver and passenger monitoring fusion method, system and related product

Abstract

The invention discloses a driver and passenger monitoring fusion method, a driver and passenger monitoring fusion system and related products, relates to the technical field of intelligent driving monitoring systems, and solves the problems that control signal lines between a CIS sensor and a vehicle-mounted system are increased, the requirement for the operational capability of the vehicle-mounted system is increased and the integration difficulty is improved due to a conventional DMS / OMS system fusion scheme. According to the technical scheme, an image sensor is adopted to achieve monitoring fusion of a driver area image and an in-cabin panoramic image, control line connection with a vehicle-mounted system is reduced, the requirement for the operational capability of the vehicle-mounted system is reduced, and the integration difficulty is lowered; a low-frame-rate in-cabin panoramic image and a high-frame-rate driver area image are obtained by switching a pixel reading mode in a frame interval, and the passenger state monitoring requirement and the high-precision driver behavior monitoring requirement are met; and the pixel reading mode is switched in the field blanking interval, so that the occurrence of waste frames / bad frames is reduced, and the fluency and the stability of an image picture are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving monitoring systems, and more specifically, to a driver and passenger monitoring fusion method, system and related products. Background Art

[0002] With the continuous development of autonomous driving technology and the acceleration of commercialization, more and more new cars on the market are equipped with DMS (Driver Monitoring System) and OMS (Occupancy Monitoring System). DMS usually uses a small-format, high-frame-rate camera installed in front of the steering wheel or on the left front to monitor whether the driver's behavior is normal. It warns the driver when there is a violation and improves driving behavior, thereby improving driving safety. Correspondingly, OMS usually uses a large-format, low-frame-rate camera installed above the center console to monitor whether the passenger's behavior and condition are abnormal.

[0003] like Figure 1 As shown in the figure, the common DMS is to install a high-frame rate CIS (Contact Image Sensor) in the front row of the car to collect the driver's facial movement information at high speed, monitor the driver's behavior through the vehicle-mounted system (MCU), and alarm the abnormal driver behavior through the vehicle-mounted alarm device to achieve DMS coverage. Figure 2 As shown in the figure, the common OMS is to install a low-frame rate CIS in the rear seat of the car at the same time, collect the behavior and action information of the rear passengers at a low speed, transmit it to the car screen through the car system, and monitor the passenger behavior through the car screen to achieve OMS coverage. In the conventional DMS and OMS system integration solution, two CIS need to be connected to the car system, resulting in an increase in the control signal lines between the CIS and the car system, increasing the computing power requirements of the car system, and the difficulty of integration.

[0004] Therefore, the present application provides a driver and passenger monitoring fusion method, system and related products to solve the above problems. Summary of the invention

[0005] The purpose of this application is to provide a driver and passenger monitoring fusion method, system and related products to solve the problem that the conventional DMS and OMS system fusion solution leads to an increase in control signal lines between the CIS sensor and the vehicle system, an increase in the computing power requirements of the vehicle system, and an increase in the difficulty of integration; this solution only carries one image sensor to achieve full coverage of the scene in the cabin, and by controlling the exposure timing and readout method of the image sensor, without increasing the clock frequency, the DMS / OMS high and low frame rate synchronous image transmission application scene fusion is achieved, the control signal line connection is reduced, the computing power requirements of the vehicle system are reduced, and the difficulty and cost of software and hardware integration are reduced.

[0006] The present application first provides a driver and passenger monitoring fusion method, which is executed based on an image sensor system, including: obtaining driver area coordinate information; determining a field blanking interval pixel readout method according to the driver area coordinate information; initiating a first exposure of the pixel matrix and obtaining a panoramic image of the cabin through FULL readout; outputting the panoramic image of the cabin through a main path data interface for monitoring passenger status; obtaining a first driver area image through cropping and addition processing of the panoramic image of the cabin, and outputting the first driver area image through a branch path data interface after preprocessing for monitoring driver behavior, the cropping and addition processing including: extracting the first driver area image from the panoramic image of the cabin according to the driver area coordinate information and the field blanking interval pixel readout method; when a frame enters the field blanking interval, initiating a second exposure of the pixel matrix and obtaining a second driver area image through a field blanking interval pixel readout method, and outputting the second driver area image through a branch path data interface after preprocessing.

[0007] In one possible implementation, a method for reading pixels in the field blanking interval is determined based on the driver area coordinate information; including: calculating the driver area size based on the starting coordinate signal and ending coordinate information in the driver area coordinate information; comparing the driver area size with the transmittable size of the field blanking interval FULL read out by the image sensor, and determining a method for reading pixels in the field blanking interval based on the comparison result.

[0008] In a possible implementation, the driver area size is compared with the transmittable size of the field blanking interval read by the image sensor FULL, and the readout method of the field blanking interval pixels is determined according to the comparison result; including: if the driver area size does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout method is switched from Full readout to ROI readout; if the driver area size exceeds the transmittable size of the field blanking interval, V2H2 addition processing is performed on the driver area size, and if the driver area size after V2H2 addition processing does not exceed If the size of the driver area after the V2H2 addition processing still exceeds the transmittable size of the field blanking interval, the driver area size will be processed by V4H4 addition; if the size of the driver area after the V4H4 addition processing does not exceed the transmittable size of the field blanking interval, the driver area coordinate information will be sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode will be switched from Full readout to V4H4 addition readout.

[0009] In a possible implementation, the preprocessing includes: filtering and noise reduction and contour information extraction processing.

[0010] In a possible implementation, the contour information extraction process is performed using a local binary method. The present application also provides a driver and passenger monitoring fusion system, including: an on-board system, an image sensor system, an on-board display screen and an on-board alarm device; the on-board system is used to obtain the driver's facial image, perform face recognition based on the driver's facial image, and re-send the instruction for obtaining the driver's facial image to the image sensor system when the face recognition fails, and obtain the driver's area coordinate information when the face recognition passes; the image sensor system is used to execute the above driver and passenger monitoring fusion method to obtain a panoramic image of the cabin and a first driver area image and a second driver area image; the on-board system is used to receive the panoramic image of the cabin output by the main path data interface, transmit the panoramic image of the cabin to the on-board display screen, receive the first driver area image and the second driver area image output by the branch path data interface, monitor the driver's behavior in real time according to the first driver area image and the second driver area image, and send an alarm message to the on-board alarm device when the driver's behavior is abnormal; the on-board display screen is used to monitor the passenger status in real time according to the panoramic image of the cabin; the on-board alarm device is used to issue an audible and / or optical alarm in response to the alarm message.

[0011] In one possible implementation, the driver's behavior is monitored in real time based on a first driver area image and a second driver area image; including: the first driver area image and the second driver area image are collectively referred to as the driver area image, the contour information of the driver area image is matched with the locally stored driver facial feature data, if the match fails, a warning is issued and new contour information is received for matching, if the match is successful, the current contour information is locked; the characteristic parts in the contour information are calibrated, and the position information and opening amplitude information of the characteristic parts are captured; the position information and opening amplitude information of the characteristic parts are input into the fatigue status evaluation model to predict whether the driver's behavior is abnormal.

[0012] The present application also provides a computer device including: a memory, and one or more processors, wherein the processor is used to execute a computer program or instruction stored in the memory, so that the computer device executes a driver and passenger monitoring fusion method as described above.

[0013] The present application also provides a computer storage medium, including a computer program or instruction, which, when executed on a computer device, enables the computer device to execute a driver and passenger monitoring fusion method as described above.

[0014] The present application also provides a computer program product. When the computer program product is run on a computer device, the computer device executes the driver and passenger monitoring fusion method as described above.

[0015] Compared with the prior art, the present application has the following beneficial effects: the present application adopts one image sensor to realize the monitoring fusion of the driver area image and the cabin panoramic image, reduces the control line connection with the vehicle-mounted system, reduces the computing power requirement of the vehicle-mounted system, and reduces the integration difficulty; switches the pixel readout mode within a frame interval to obtain a low-frame rate cabin panoramic image and a high-frame rate driver area image, meeting the passenger status monitoring needs and high-precision driver behavior monitoring needs; switches the pixel readout mode in the field blanking interval to reduce the occurrence of waste frames / bad frames, and ensures the smoothness and stability of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 This is a common DMS structure diagram; Figure 2 This is a common OMS structure diagram; Figure 3 A flowchart of a driver and passenger monitoring fusion method provided in an embodiment of the present application; Figure 4It is a schematic diagram of clipping addition and field blanking readout in the case of ROI readout; Figure 5 It is a schematic diagram of clipping addition and field blanking readout in the case of V2H2 readout; Figure 6 It is a schematic diagram of clipping addition and field blanking readout in the case of V4H4 readout; Figure 7 A schematic diagram of the main path data interface and the branch path data interface images obtained by switching exposure readout; Figure 8 Schematic diagram of dynamically switching exposure readout for a conventional image sensor; Fig. 9 A schematic diagram of dynamically switching exposure readout of the image sensor of the present application; Fig.10 A schematic diagram of a driver and passenger monitoring fusion system provided in an embodiment of the present application; Fig.11 Workflow diagram of the fusion system for driver and passenger monitoring; Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] Hereinafter, the term "include" or "may include" used in various embodiments of the present application indicates the presence of the function, operation or element applied for, and does not limit the increase of one or more functions, operations or elements. In addition, as used in various embodiments of the present application, the terms "include", "have" and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or the possibility of increasing one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items.

[0018] The expressions (such as "first", "second", etc.) used in various embodiments of the present application may modify various constituent elements in various embodiments, but may not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used for the purpose of distinguishing one element from other elements. For example, the first user device and the second user device indicate different user devices, although both are user devices. For example, without departing from the scope of various embodiments of the present application, the first element may be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0019] The terms used in the various embodiments of the present application are only used for the purpose of describing specific embodiments and are not intended to limit the various embodiments of the present application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the meanings commonly understood by ordinary technicians in the field to which the various embodiments of the present application belong. The terms (such as those defined in a generally used dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the present application.

[0020] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

[0021] See also Figure 3 As shown, Figure 3 A flowchart of a driver and passenger monitoring fusion method provided in an embodiment of the present application. The driver and passenger monitoring fusion method is executed based on an image sensor system, including: obtaining driver area coordinate information; determining a field blanking interval pixel readout method according to the driver area coordinate information; initiating a first exposure of the pixel matrix and obtaining a panoramic image of the cabin through FULL readout; outputting the panoramic image of the cabin through a main path data interface for monitoring passenger status; obtaining a first driver area image through cropping and addition processing of the panoramic image of the cabin, and outputting the first driver area image through a branch path data interface after preprocessing for monitoring driver behavior, wherein the cropping and addition processing includes: extracting the first driver area image from the panoramic image of the cabin according to the driver area coordinate information and the field blanking interval pixel readout method; when the frame enters the field blanking interval, initiating a second exposure of the pixel matrix and obtaining a second driver area image through a field blanking interval pixel readout method, and outputting the second driver area image through a branch path data interface after preprocessing.

[0022] Specifically, the image sensor system obtains the driver area coordinate information from the vehicle control system, and determines the switching mode of the pixel readout method in the field blanking interval of the frame based on this information. The exposure readout control module initiates the first exposure readout control of the pixel matrix, and obtains a panoramic image of the cabin through the FULL (full size) readout method. The panoramic image of the cabin is processed through two data paths respectively. The panoramic image of the cabin is not processed in other special ways on the main path, and is directly output through the main path data interface. The panoramic image of the cabin on the branch path is passed to the cropping and addition module, and the zoom factor, cropping position and size are determined by the aforementioned driver area coordinate information and the field blanking interval pixel readout method, and the driver area image is extracted from the panoramic image of the cabin to obtain the first driver area image. Then, the pre-processing module performs filtering and noise reduction, contour information extraction and other processing to reduce the amount of calculation for the subsequent vehicle system to monitor the driver's behavior, and finally outputs it through the branch path data interface. After obtaining the panoramic image of the cabin and the first driver area image, when the frame enters the field blanking interval, the exposure readout control module initiates the second exposure readout control of the frame to the pixel matrix unit, and the readout mode is dynamically switched from FULL readout to the field blanking interval pixel readout mode obtained previously, so as to realize fast exposure readout of only the driver area and obtain the second driver area image. The second driver area image is preprocessed through the branch path. Since the image read out of the second exposure only contains the driver area, the cropping and addition module does not process the second driver area image at this time, but directly performs filtering and noise reduction, contour information extraction and other processing through the pre-module, extracts key information, and finally outputs it through the branch path data interface. The low frame rate panoramic image of the cabin received by the main path data interface can be used for passenger status monitoring of the subsequent vehicle display screen. The contour information of the high frame rate driver area image (first driver area image, second driver area image) output by the branch path data interface can be used for driver behavior monitoring of the vehicle system.

[0023] The improvements of the present application are that, first, the number of image sensors is reduced from two to one, which reduces the number of interface requirements for the vehicle-mounted system, simplifies the difficulty of wiring with the vehicle-mounted system, reduces the computing power requirements for the vehicle-mounted system, and realizes stable integrated coverage of DMS and OMS; second, two exposures are set within one frame interval to obtain a panoramic image of the cabin and two driver area images, and high-frequency acquisition is performed on the driver area and low-frequency acquisition is performed on the panoramic view of the cabin, ensuring that the passenger status can be monitored while accurately tracking the driver's behavior to avoid dangerous driving; third, switching the readout mode in the field blanking interval can reduce the waste frames / bad frames existing in traditional exposure and readout switching, ensuring the smoothness and stability of the image.

[0024] It needs to be explained that field blanking (VBlank) refers to the idle period between the end of one frame of data transmission and the start of the next frame of data transmission, during which the functional module does not operate. The transmittable size of the field blanking interval refers to the frame rate calculation based on the current readout method. The frame rate is fixed, that is, the time of one frame is fixed, and the image data frame and transmission rate of one frame are fixed, then the time required for the transmission of one frame of image data can be calculated. The field blanking interval length can be obtained by subtracting the time required for the transmission of one frame of image data from the frame time. The amount of data that can be transmitted during this time is the transmittable size of the field blanking interval.

[0025] Assuming that the driver area is fixed and the sensor model is fixed, the readout method of the field blanking interval pixels can be directly preset according to the fixed driver area coordinate information and the fixed sensor model to ensure that the image of the driver area can be obtained after the field blanking interval readout method is switched. However, the actual situation is more complicated. The driver areas of different models may be different, the image sizes of FULL readouts of image sensors of different models are different, and the transmittable sizes of the field blanking intervals are also different. Therefore, it is difficult to meet the usage requirements in complex scenarios by adopting a single preset field blanking interval pixel readout method. To this end, the present application proposes a method for determining the field blanking interval pixel readout method to meet the usage requirements in complex scenarios.

[0026] In one possible implementation, a method for reading pixels in the field blanking interval is determined based on the driver area coordinate information; including: calculating the driver area size based on the starting coordinate signal and ending coordinate information in the driver area coordinate information; comparing the driver area size with the transmittable size of the field blanking interval FULL read out by the image sensor, and determining a method for reading pixels in the field blanking interval based on the comparison result.

[0027] Furthermore, the driver area size is compared with the transmittable size of the field blanking interval read out by the image sensor in FULL, and the readout method of the field blanking interval pixels is determined according to the comparison result; including: if the driver area size does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout method is switched from Full readout to ROI readout; if the driver area size exceeds the transmittable size of the field blanking interval, V2H2 addition processing is performed on the driver area size, and if the driver area size after V2H2 addition processing does not exceed the field blanking interval, If the size of the driver area after the V2H2 addition processing still exceeds the transmittable size of the field blanking interval, the driver area size will be processed by V4H4 addition; if the size of the driver area after the V4H4 addition processing does not exceed the transmittable size of the field blanking interval, the driver area coordinate information will be sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode will be switched from Full readout to V4H4 addition readout.

[0028] Specifically, the driver area size is compared with the transmittable size of the field blanking interval read out by the image sensor FULL, and the readout method of the field blanking interval pixels is determined according to the comparison result. When the driver area size is less than or equal to the transmittable size of the field blanking interval read out by the image sensor FULL, it means that the field blanking interval read out by FULL can transmit the image of the driver area. At this time, the image of the driver area can be read out by simply changing the pixel readout method to ROI (region of interest) readout. When the driver area size is larger than the transmittable size of the field blanking interval read out by the image sensor FULL, it means that the field blanking interval read out by FULL cannot transmit the image of the driver area. At this time, it is necessary to change the pixel readout method, such as using V2H2 addition (2 rows and 2 columns of different colors merged) readout, V4H4 addition (4 rows and 4 columns of different colors merged) readout, etc., to reduce the number of pixels read out and output the driver area image at a lower resolution.

[0029] It can be understood that by switching from Full readout to ROI readout or V2H2 addition readout or V4H4 addition readout, the driving area can be read out for a second exposure in a very short time. This is equivalent to completing two driving area image readouts at the same time each time the panoramic image of the cabin is read out, thereby achieving high frame rate readout of the driver area image, meeting the DMS high frame rate motion detection requirements, and further completing DOMS (Driver and Occupancy Monitoring System) fusion.

[0030] Combine the following Figure 4-6 The workflow of the image sensor system is described in detail for three possible readout methods (ROI readout, V2H2 readout, and V4H4 readout).

[0031] When the image sensor system is started, after receiving the driver area coordinate information pre-calibrated by the vehicle-mounted system, the driver area size is calculated through the start and end coordinate information in the driver area coordinate information, and the driver area size is compared with the transmittable size of the field blanking interval of the current image sensor readout mode (Full), and the field blanking interval pixel readout mode is preset according to the comparison result; at the same time, the driver area coordinate information and the preset field blanking interval pixel readout mode are sent to the cropping and addition module. The driver area coordinate information can ensure that the driver area image is accurately cropped from the panoramic image in the cabin, and the field blanking interval pixel readout mode can ensure that the appropriate addition action is matched so that the field blanking interval can fully transmit the driver area image, thereby ensuring the continuity of the image.

[0032] The three readout methods of the field blanking interval are as follows: Case 1: If Figure 4 As shown, Figure 4 The following is a schematic diagram of clipping addition and field blanking readout in the case of ROI readout. Figure 4 As shown in the lower part, if the current driver area size does not exceed the transmittable size in the field blanking interval, the exposure readout control module will send the driver area coordinate information to the pixel matrix at the beginning of the field blanking of the current frame, and the readout mode will switch from Full readout to ROI readout to ensure that the driver area image is obtained by double exposure readout in the field blanking interval. Figure 4 As shown in the upper part, the exposure readout control module will also send the driver area coordinate information and the field blanking interval pixel readout method to the cropping and addition module to determine the cropping start position and addition information (ROI readout is not added). The cropping and addition function will act on the cabin panoramic image read out by a single exposure FULL, and the driver area image will be obtained by cropping and adding.

[0033] Case 2: If Figure 5 As shown, Figure 5 Schematic diagram of clipping addition and field blanking readout in V2H2 readout. Figure 5 As shown in the lower part, if the current driver area size exceeds the transmittable size of the field blanking interval, the size of the driver area after V2H2 addition is compared with the transmittable size of the field blanking interval. If the driver area size after V2H2 addition does not exceed the transmittable size of the field blanking interval, the readout control module will send the driver area coordinate information to the pixel matrix at the beginning of the field blanking of the current frame, and the readout mode will be switched from Full readout to V2H2 addition readout to ensure that the driver area image is obtained by double exposure readout in the field blanking interval. Figure 5 As shown in the upper part, the exposure readout control module will also send the driver area coordinate information and the field blanking interval pixel readout method to the cropping and addition module to determine the cropping start position and addition information (V2H2 readout is processed as V2H2 addition). The cropping and addition function will act on the cabin panoramic image read out by a single exposure FULL, and the driver area image will be obtained by cropping and addition.

[0034] Case 3: If Figure 6 As shown, Figure 6 Schematic diagram of clipping addition and field blanking readout in V4H4 readout. Figure 6 As shown in the lower part, if the current driver area size exceeds the transmittable size of the field blanking interval, the size of the driver area after V2H2 addition is compared with the transmittable size of the field blanking interval. If the driver area size after V2H2 addition still exceeds the transmittable size of the field blanking interval, the size of the driver area after V4H4 addition is compared with the transmittable size of the field blanking interval. If the driver area size after V4H4 addition does not exceed the transmittable size of the field blanking interval, the readout control module will send the driver area coordinate information to the pixel matrix at the beginning of the field blanking of the current frame, and the readout mode will be switched from Full readout to V4H4 addition readout to ensure that the driver area image is obtained by double exposure readout in the field blanking interval. Figure 6 As shown in the upper part, the exposure readout control module will also send the driver area coordinate information and the field blanking interval pixel readout method to the cropping and addition module to determine the cropping start position and addition information (V4H4 readout is processed by V4H4 addition). The cropping and addition function will act on the cabin panoramic image read out by a single exposure FULL, and the driver area image will be obtained by cropping and addition.

[0035] See also Figure 7 As shown, Figure 7 The following is a schematic diagram of the main path data interface and branch path data interface images obtained by switching exposure readout. In the figure, XVS is the vertical synchronization signal, N Frame is the Nth frame, and N+1 Frame is the N+1th frame. A low frame rate image sensor can simultaneously transmit two channels of high and low frame rate image data.

[0036] It should be noted that in addition to ROI readout, V2H2 readout, and V4H4 readout, the field blanking interval pixel readout method can also be other common pixel merging modes, such as NxN Binning: where "N" can be any positive integer, indicating that N pixels are merged into one pixel in the horizontal and vertical directions. Another example is discarding some pixels, irregularly merging pixels (1x2, 2x3, etc.), combining irregularly merging pixels with discarding some pixels, etc. ROI, V2H2, and V4H4 do not constitute a restriction on the field blanking interval pixel readout method. Any field blanking interval pixel readout method that ensures that the driver area size does not exceed the transmittable size of the field blanking interval can be used as the field blanking interval pixel readout method of the present application.

[0037] It can be understood that in the driver and passenger monitoring fusion method proposed in the present application, the image sensor system needs to dynamically switch the image exposure readout mode in the working mode to complete the acquisition of three image data within one frame time. The main path outputs a low frame rate cabin panoramic image for OMS monitoring, and the branch path outputs two high frame rate driver area images for DMS monitoring, thereby realizing the simultaneous output of high and low frame rate images.

[0038] like Figure 8 As shown, Figure 8 The following is a schematic diagram of the dynamic switching of exposure readout of a traditional image sensor. When a traditional image sensor dynamically switches exposure readout, because the exposure of the frame readout pixel data has already started in the previous frame, the dynamic switching of the exposure readout mode will cause the pixel exposure and pixel readout methods of the previous and next frames to not match, resulting in an abnormal image frame after the mode switch, causing the image sensor to produce discontinuous output images, and bad frames / wasted frames appearing on the way, affecting the subsequent driver behavior monitoring and causing false alarms in the driver behavior monitoring. Please refer to Fig. 9 As shown, Fig. 9 This is a schematic diagram of the dynamic switching exposure readout of the image sensor of the present application. The present application proposes to dynamically switch the exposure readout in the field blanking interval, which effectively avoids the occurrence of bad frames / wasted frames between frames, ensures the continuity of the output image, and improves the accuracy of the in-cabin monitoring.

[0039] In one possible implementation, preprocessing includes: filtering and noise reduction and contour information extraction processing. Specifically, in the image sensor system of the present application, two data paths with different frame rates, the main path and the branch path, are obtained to meet the OMS and DMS monitoring requirements respectively. In order to further reduce the difficulty of data reception and calculation of the back-end vehicle-mounted system, the present application also proposes to pre-install a preprocessing module inside the image sensor system, pre-process the driver area image, extract key information after filtering and noise reduction, and simplify the transmission data to facilitate the back-end vehicle-mounted system to monitor the driver's behavior.

[0040] Furthermore, the contour information extraction process is performed using a local binary method. Specifically, driver behavior monitoring mainly requires the collection of features such as the driver's eyes, nose, mouth and facial contours. The present application can use LBP (local binary pattern) to extract facial features of the face, and utilize the LBP operator to have feature invariance to the overall illumination. When the light in the cockpit changes, the contrast can be maintained. The LBP feature can well cope with overall illumination interference and local weak interference, and accurately extract facial feature information. Specifically, a 3x3 pixel matrix is ​​defined on the branch path, and the grayscale value of the center pixel of this matrix is ​​used as the threshold, and the grayscale values ​​of the surrounding 8 pixels are compared with it. If the grayscale value of the surrounding pixels is greater than the grayscale value of the center pixel, the grayscale value of the pixel is marked as 1, otherwise it is marked as 0. After all the 8 surrounding pixels in the 3x3 pixel matrix are compared and marked, an 8-bit binary number can be obtained. This value is the LBP value of the center pixel of the 3x3 pixel matrix, and this LBP value can reflect the texture information of the matrix range. The driver area image is scanned sequentially using the LBP operator to obtain the feature information of the driver area image.

[0041] See also Fig.10 As shown, Fig.10 A schematic diagram of a driver and passenger monitoring fusion system provided in an embodiment of the present application. The system includes: an on-board system, an image sensor system, an on-board display screen, and an on-board alarm device; the on-board system is used to obtain a driver's facial image, perform face recognition based on the driver's facial image, and resend an instruction for obtaining the driver's facial image to the image sensor system when the face recognition fails, and obtain the driver's area coordinate information when the face recognition passes; the image sensor system is used to execute the above driver and passenger monitoring fusion method to obtain a cabin panoramic image and a first driver area image and a second driver area image; the on-board system is used to receive the cabin panoramic image output by the main path data interface, transmit the cabin panoramic image to the on-board display screen, receive the first driver area image and the second driver area image output by the branch path data interface, monitor the driver's behavior in real time according to the first driver area image and the second driver area image, and send an alarm message to the on-board alarm device when the driver's behavior is abnormal; the on-board display screen is used to monitor the passenger status in real time according to the cabin panoramic image; the on-board alarm device is used to send an audible and / or optical alarm in response to the alarm message.

[0042] See also Fig.11 As shown, Fig.11 The working flow diagram of the driver and passenger monitoring fusion system. When the driver and passenger monitoring fusion system works: The vehicle-mounted system is started, the driver matching and identification is performed and the driver area coordinate information is stored. After successful identification, the image sensor system is controlled to start and the driver area coordinate information is sent to the image sensor system.

[0043] The exposure readout control module of the image sensor system calculates the driver area size based on the driver area coordinate information and compares it with the transmittable size of the field blanking area under FULL readout, determines the field blanking interval pixel readout method of the image sensor, and sends the driver area coordinate information and the field blanking interval pixel readout method to the back-end cropping and addition module.

[0044] After the pixel matrix is ​​FULL read out, the panoramic image of the cabin is read out. On the one hand, the panoramic image of the cabin enters the main path: the panoramic image of the cabin is transmitted to the vehicle-mounted system through the main path data interface; the vehicle-mounted system sends the panoramic image of the cabin to the vehicle-mounted screen for passenger status monitoring (OMS); after the transmission of a frame of the panoramic image of the cabin in the main path is completed, the image sensor system enters the field blanking interval. On the other hand, the panoramic image of the cabin enters the branch path: the panoramic image of the cabin is transmitted to the cropping and addition module, and the corresponding cropping and addition processing is performed according to the driver area coordinate information obtained from the exposure readout module and the field blanking interval pixel readout method to obtain the first driver area image; the first driver area image is transmitted to the preprocessing module for noise reduction filtering and contour information extraction processing to improve the picture clarity and reduce the amount of data for driver behavior monitoring by the vehicle-mounted system; the preprocessed first driver area image is transmitted to the vehicle-mounted system through the branch path data interface; the vehicle-mounted system has a built-in MCU for driver behavior monitoring, and controls the vehicle-mounted alarm device to alarm (DMS) when abnormal driver behavior is detected; after the transmission of a frame of the driver area image in the branch path is completed, the image sensor system enters the field blanking interval.

[0045] The image sensor system enters the field blanking interval: the exposure readout control module controls the pixel matrix to perform a second pixel exposure readout according to the previously obtained driver area coordinate information and the pixel readout method during the field blanking period, so as to obtain a second driver area image; the second driver area image is transmitted to the preprocessing module for noise reduction filtering and contour information extraction processing to improve image clarity and reduce the amount of data for driver behavior monitoring by the on-board system; the preprocessed second driver area image is transmitted to the on-board system through the branch path data interface; the on-board system has a built-in MCU to monitor the driver behavior and controls the on-board alarm device (DMS) to alarm when abnormal driver behavior is detected.

[0046] Specifically, the driver starts the vehicle, controls the image sensor system to collect the driver's facial image at a lower frame rate, and performs driver face recognition through the built-in MCU. If the recognition is abnormal (such as no portrait information is collected, the collected portrait information does not match the pre-recorded portrait information, etc.), the abnormal information is prompted on the vehicle screen and the image sensor system is controlled again to collect the driver's facial image. After the recognition is passed, the driver area coordinate information is extracted and sent to the image sensor system. The image sensor system defaults to FULL reading. Through the received driver area coordinate information, the driver area size is calculated, that is, the size occupied by the driver area in the panoramic image in the cabin, and compared with the field blanking interval size transmittable size under the current image sensor readout mode, the field blanking interval pixel readout mode is determined. The image sensor system starts working, Full readout->ROI / V2H2 / V4H4 readout cyclic switching, and obtains two high- and low-frame-rate transmission images (low-frame-rate panoramic images of the cabin in the main path, and high-frame-rate images of the driver's area in the branch path). The two data are transmitted to the back-end vehicle system through the two high- and low-frame-rate data interfaces (main path data interface, branch path data interface) carried by the image sensor system. The vehicle system obtains a low-frame-rate panoramic image of the cabin and transmits it to the vehicle screen for real-time monitoring of the vehicle environment and passenger status to realize the OMS function. At the same time, the vehicle system obtains a high-frame-rate driver's area image and transmits it to the built-in MCU for driver behavior monitoring. When abnormal driving behavior is detected, an early warning is issued through the vehicle alarm device (such as a speaker) to realize the DMS function.

[0047] In one possible implementation, the driver's behavior is monitored in real time based on a first driver area image and a second driver area image; including: the first driver area image and the second driver area image are collectively referred to as the driver area image, the contour information of the driver area image is matched with the locally stored driver facial feature data, if the match fails, a warning is issued and new contour information is received for matching, if the match is successful, the current contour information is locked; the characteristic parts in the contour information are calibrated, and the position information and opening amplitude information of the characteristic parts are captured; the position information and opening amplitude information of the characteristic parts are input into the fatigue status evaluation model to predict whether the driver's behavior is abnormal.

[0048] Specifically, the driver area image is compared and matched with the driver's facial feature data pre-stored locally. If the match fails, a warning is issued and a new input image is extracted to continue matching. If the match is completed, the current facial features are locked and the driver fatigue monitoring continues. Taking facial monitoring as an example, the characteristic parts such as eyes and mouth are calibrated, and the position, opening range and other information of the eyes and mouth at each moment are captured. Through the statistics and analysis of this information, a fatigue status evaluation model is established in combination with multiple representative fatigue status evaluation indicators to realize the monitoring and judgment of the driver's mental state at any time. At the same time, an alarm reminder is issued according to the monitoring results, and the driver is prompted to pay attention to driving safety through the voice of the onboard alarm equipment to complete the DMS real-time monitoring.

[0049] In summary, the present application uses an image sensor to realize the monitoring fusion of the driver area image and the panoramic image in the cabin, reduces the control line connection with the vehicle-mounted system, reduces the computing power requirements of the vehicle-mounted system, and reduces the difficulty of integration; switches the pixel readout mode within a frame interval to obtain a low-frame rate panoramic image in the cabin and a high-frame rate driver area image, meeting the needs of passenger status monitoring and high-precision driver behavior monitoring; switches the pixel readout mode in the field blanking interval to reduce the occurrence of waste frames / bad frames, and ensures the smoothness and stability of the image.

[0050] See also Fig.12 As shown, Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes: a memory, and one or more processors, wherein the processor is used to execute a computer program or instruction stored in the memory, so that the computer device executes a driver and passenger monitoring fusion method as described above.

[0051] The computer device 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1222 (for example, one or more processors) and memory 1232, and one or more storage media 1230 (for example, one or more mass storage devices) storing application programs 1242 or data 1244. Among them, the memory 1232 and the storage medium 1230 may be temporary storage or permanent storage. The program stored in the storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device. Furthermore, the central processing unit 1222 may be configured to communicate with the storage medium 1230 to execute a series of instruction operations in the storage medium 1230 on the computer device 1200.

[0052] The computer device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input data interfaces 1258, and / or, one or more operating systems 1241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0053] In an embodiment of the present application, the central processor 1222 can be used to execute the above-mentioned driver and passenger monitoring fusion method.

[0054] An embodiment of the present application also provides a computer storage medium, including a computer program or instructions. When the computer program or instructions are executed on a computer device, the computer device executes a driver and passenger monitoring fusion method as described above.

[0055] An embodiment of the present application also provides a computer program product. When the computer program product is executed on a computer device, the computer device executes a driver and passenger monitoring fusion method as described above.

[0056] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A driver and passenger monitoring fusion method, characterized in that: Image sensor-based system execution includes: Get the driver area coordinate information; Determine a pixel readout method in a field blanking interval according to the driver area coordinate information; Initiate the first exposure of the pixel matrix and obtain a panoramic view of the cabin through FULL readout; Outputting the cabin panoramic image through the main path data interface for monitoring passenger status; The first driver area image is obtained by cutting and adding the panoramic image in the cabin, and the first driver area image is output through the branch path data interface after preprocessing for monitoring the driver's behavior, wherein the cutting and adding processing includes: extracting the first driver area image from the panoramic image in the cabin according to the driver area coordinate information and the field blanking interval pixel reading method; When the frame enters the field blanking interval, a second exposure is initiated for the pixel matrix and a second driver area image is obtained by a field blanking interval pixel readout method, and the second driver area image is output through a branch path data interface after preprocessing.

2. A driver and passenger monitoring fusion method according to claim 1, characterized in that: Determining a pixel readout method in a field blanking interval according to the driver area coordinate information; comprising: Calculate the driver area size according to the starting coordinate signal and the ending coordinate information in the driver area coordinate information; The driver area size is compared with the transmittable size of the field blanking interval read out by the image sensor FULL, and the readout method of the field blanking interval pixels is determined according to the comparison result.

3. A driver and passenger monitoring fusion method according to claim 2, characterized in that: Comparing the driver area size with the transmittable size of the field blanking interval read out by the image sensor FULL, and determining the readout method of the field blanking interval pixels according to the comparison result; comprising: If the size of the driver area does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode is switched from Full readout to ROI readout; If the size of the driver area exceeds the size that can be transmitted in the field blanking interval, V2H2 addition processing is performed on the driver area size. If the size of the driver area after the V2H2 addition processing does not exceed the size that can be transmitted in the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode is switched from Full readout to V2H2 addition readout; If the size of the driver area after the V2H2 addition processing still exceeds the transmittable size of the field blanking interval, the V4H4 addition processing is performed on the driver area size. If the size of the driver area after the V4H4 addition processing does not exceed the transmittable size of the field blanking interval, the driver area coordinate information is sent to the pixel matrix at the beginning of the field blanking interval of the current frame, and the readout mode is switched from Full readout to V4H4 addition readout.

4. A driver and passenger monitoring fusion method according to claim 1, characterized in that: The preprocessing includes: filtering and noise reduction and contour information extraction processing.

5. A driver and passenger monitoring fusion method according to claim 4, characterized in that: The contour information extraction process is performed using a local binary method.

6. A driver and passenger monitoring fusion system, characterized in that: A driver and passenger monitoring fusion system comprising: In-vehicle systems, image sensor systems, in-vehicle displays and in-vehicle warning devices; The vehicle-mounted system is used to obtain a facial image of the driver, perform face recognition based on the facial image of the driver, resend an instruction for obtaining the facial image of the driver to the image sensor system when the face recognition fails, and obtain the driver area coordinate information when the face recognition passes; The image sensor system is used to execute a driver and passenger monitoring fusion method as described in any one of claims 1 to 5 to obtain a cabin panoramic image and a first driver area image and a second driver area image; The vehicle-mounted system is used to receive the panoramic image of the cabin output by the main path data interface, transmit the panoramic image of the cabin to the vehicle-mounted display screen, receive the first driver area image and the second driver area image output by the branch path data interface, monitor the driver's behavior in real time according to the first driver area image and the second driver area image, and send an alarm message to the vehicle-mounted alarm device when the driver's behavior is abnormal; The vehicle-mounted display screen is used to monitor the passenger status in real time based on the panoramic image in the cabin; The vehicle-mounted alarm device is used to emit sound and / or light alarm in response to the alarm information.

7. A driver and passenger monitoring fusion system according to claim 6, characterized in that: Real-time monitoring of driver behavior based on a first driver area image and a second driver area image; including: The first driver area image and the second driver area image are collectively referred to as the driver area image, and the contour information of the driver area image is matched with the driver's facial feature data pre-stored locally. If the match fails, a warning is issued and new contour information is received for matching. If the match succeeds, the current contour information is locked; The characteristic parts in the contour information are calibrated, and the position information and the opening amplitude information of the characteristic parts are captured; The position information and opening amplitude information of the characteristic parts are input into the fatigue state evaluation model to predict whether the driver's behavior is abnormal.

8. A computer device, characterized in that: include: A memory, one or more processors, wherein the processor is used to execute a computer program or instruction stored in the memory so that the computer device executes a driver and passenger monitoring fusion method as described in any one of claims 1-5.

9. A computer storage medium, characterized in that It comprises a computer program or an instruction, which, when executed on a computer device, enables the computer device to execute a driver and passenger monitoring fusion method as claimed in any one of claims 1 to 5.

10. A computer program product, characterized in that When the computer program product runs on a computer device, the computer device is enabled to execute a driver and passenger monitoring fusion method as described in any one of claims 1 to 5.

Images