A method, device and equipment for recognizing driver state and a storage medium

Abstract

This invention discloses a method, apparatus, device, and storage medium for driver state recognition. The method includes: performing real-time eye recognition on the current driver to obtain an eye region; when a first light ray is projected onto the eye region, acquiring an image of the eye region based on a first image acquisition device corresponding to the first light ray to obtain a black and white image; when a second light ray is projected onto the remaining area inside the cockpit excluding the eye region, acquiring an image of the remaining area based on a second image acquisition device corresponding to the second light ray to obtain a color image; fusing the black and white image and the color image to obtain a fused image; and determining the current driver's driving state based on the fused image. This invention reduces light interference to the driver by setting two types of light to supplement the lighting inside the cockpit and acquiring images after supplemental lighting, ensuring driving safety while maintaining the accuracy of driver state recognition.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, specifically to a method, apparatus, device, and storage medium for driver state recognition. Background Technology

[0002] With the rapid development of intelligent driving, occupant monitoring inside the intelligent cockpit is particularly important for driving safety. Algorithms such as DMS (Driver Monitoring System) can monitor for fatigued driving and dangerous driving behaviors such as closing eyes and not looking straight ahead, thereby reducing the probability of accidents caused by these behaviors.

[0003] In existing technologies, DMS typically uses infrared illumination to acquire facial and background images. However, infrared illumination produces only black-and-white grayscale images, which is not conducive to algorithms' fine differentiation and recognition of faces and backgrounds. Alternatively, internal white light sources can be used to illuminate the cabin, but white light illumination can easily cause glare on the windshield during nighttime driving, interfering with the driver's vision and potentially leading to traffic accidents. Therefore, obtaining easily processed images without interfering with the driver's vision is a pressing problem that needs to be solved. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to improve the accuracy of driver status recognition and analysis while improving driving safety.

[0005] To address at least one of the aforementioned technical problems, this invention discloses a method, apparatus, device, and storage medium for driver status recognition.

[0006] According to one aspect of this disclosure, a method for driver state recognition is provided, comprising:

[0007] Real-time eye recognition is performed on the current driver to obtain the eye region;

[0008] When a first ray of light is projected onto the eye region, an image of the eye region is acquired based on a first image acquisition device corresponding to the first ray of light, resulting in a black and white image; the first ray of light is matched with the human eye's perception of light;

[0009] When the second light beam is projected onto the remaining area of ​​the cockpit excluding the eye area, an image is acquired from the remaining area based on the second image acquisition device corresponding to the second light beam to obtain a color image.

[0010] The black-and-white image and the color image are fused to obtain a fused image;

[0011] The current driver's driving status is determined based on the fused image.

[0012] In some possible embodiments, when the first light ray is projected onto the eye region, acquiring an image of the eye region based on a first image acquisition device corresponding to the first light ray to obtain a black and white image includes:

[0013] Based on the grid partitioning results, region detection is performed within the cockpit to obtain a first grid partition corresponding to the eye region; the grid partitioning results are determined based on the partitioning processing performed within the cockpit.

[0014] The first projection attribute of the light source projection point is determined based on the first grid partition; the first projection attribute includes the target projection angle and the target projection direction.

[0015] Based on the first projection angle, the first projection direction, and the first image acquisition device, an image of the eye region is acquired to obtain a black and white image.

[0016] In some possible embodiments, the step of acquiring an image of the eye region based on the first projection angle, the first projection direction, and the first image acquisition device to obtain a black and white image includes:

[0017] Based on the first projection angle and the first projection direction, the first light ray is projected onto the first grid partition;

[0018] The first image acquisition device acquires an image of the first grid partition on which the first light has been projected, thereby obtaining the black and white image.

[0019] In some possible embodiments, when the second light ray is projected onto the remaining area inside the cockpit excluding the eye area, acquiring an image of the remaining area based on a second image acquisition device corresponding to the second light ray to obtain a color image includes:

[0020] Based on the grid partitioning results, region detection is performed in the cockpit to obtain a second grid partitioning corresponding to the remaining region;

[0021] Based on preset partitioning rules, the second grid partition is divided into multiple grid units; each grid unit includes multiple grids.

[0022] The second image acquisition device acquires images from the multiple grid units to obtain a color image.

[0023] In some possible embodiments, the step of acquiring images from the plurality of grid units based on the second image acquisition device to obtain a color image includes:

[0024] A second projection attribute of the light source projection point is determined based on the plurality of grid cells; the second projection attribute includes a plurality of projection angles corresponding to the plurality of grid cells, and a plurality of projection directions corresponding to the plurality of grid cells;

[0025] Based on the multiple projection angles, the multiple projection directions, and the second image acquisition device, images are acquired from the multiple grid units to obtain color images.

[0026] In some possible embodiments, the step of acquiring images from the plurality of grid units based on the plurality of projection angles, the plurality of projection directions, and the second image acquisition device to obtain a color image includes:

[0027] Based on the projection angle and projection direction corresponding to each of the grid cells, the second light ray is sequentially projected onto each of the grid cells;

[0028] Based on the second image acquisition device, images are acquired for each of the grid units that have been projected with the second light, to obtain multiple images to be stitched together corresponding to each of the grid units;

[0029] The multiple images to be stitched together are processed to obtain the color image.

[0030] In some possible embodiments, the image fusion of the black-and-white image and the color image to obtain a fused image includes:

[0031] Based on the black-and-white image and the color image, repeating region detection is performed to obtain repeating regions in the color image that are repeated in the black-and-white image;

[0032] The repeated regions in the color image are deleted to obtain the color image to be fused;

[0033] The black-and-white image and the color image to be fused are fused to obtain the fused image.

[0034] In some possible embodiments, determining the current driver's driving state based on the fused image includes:

[0035] Based on the fused image, the current driver's posture and eye opening / closing are detected, and posture detection results and eye opening / closing detection results are obtained respectively.

[0036] Based on the posture detection results and the opening / closing detection results, the current driving state of the driver is determined.

[0037] In some possible embodiments, after determining the current driver's driving state based on the opening / closing detection result, the method includes:

[0038] If the posture detection result indicates that the current driver's posture does not conform to the preset driving posture, or if the opening and closing detection result indicates that the current driver's eye opening and closing degree is lower than the preset opening and closing threshold, a driving takeover command is generated.

[0039] Based on the aforementioned driver takeover command, the vehicle is controlled to enter autonomous driving mode.

[0040] According to a second aspect of this disclosure, a driver state recognition apparatus is provided, the apparatus comprising:

[0041] The eye recognition module is used to perform real-time eye recognition on the current driver to obtain the eye area;

[0042] The black and white image acquisition module acquires an image of the eye region based on a first image acquisition device corresponding to the first light ray when the first light ray is projected onto the eye region, thereby obtaining a black and white image; the first light ray is matched with the human eye's perception of light;

[0043] A color image acquisition module is used to acquire an image of the remaining area of ​​the cockpit, excluding the eye area, based on a second image acquisition device corresponding to the second light beam, to obtain a color image when the second light beam is projected onto the remaining area of ​​the cockpit.

[0044] An image fusion module is used to fuse the black-and-white image and the color image to obtain a fused image;

[0045] The state analysis module is used to determine the current driving state of the driver based on the fused image.

[0046] According to a third aspect of this disclosure, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the driver state recognition method as described above.

[0047] According to a fourth aspect of this disclosure, a computer storage medium is provided that stores at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by a processor to implement the driver state recognition method as described above.

[0048] Implementing this invention has the following beneficial effects:

[0049] This invention, after determining the driver's eye area, projects a first light beam onto that area and a second light beam onto the remaining areas inside the cockpit. By projecting different light beams to different areas to supplement the cockpit lighting, the interference of the supplementary lighting source on the driver's vision can be reduced, thereby ensuring driving safety. Simultaneously, by segmenting and projecting different light beams to different areas, the image quality of the acquired cockpit interior images can be guaranteed, thereby improving the accuracy of driver state analysis and recognition. Analyzing the driver's driving state based on the fused image can avoid dangers caused by poor driver condition, thus ensuring vehicle driving safety. Therefore, this invention achieves the technical effect of improving the quality of acquired images while minimizing interference with the driver's vision. Attached Figure Description

[0050] To more clearly illustrate the technical solution of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the driver state recognition method provided in an embodiment of the present invention.

[0052] Figure 2 A schematic diagram of the hardware structure of the integrated bay-frame domain controller is provided for embodiments of the present invention;

[0053] Figure 3 This is a flowchart illustrating the process of determining the first projection attribute according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the process for acquiring black and white images provided in an embodiment of the present invention;

[0055] Figure 5 This is a flowchart illustrating the process of dividing a grid unit according to an embodiment of the present invention.

[0056] Figure 6 This is a flowchart illustrating the process of determining the second projection attribute according to an embodiment of the present invention;

[0057] Figure 7 This is a schematic diagram of the process for acquiring color images provided in an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the image fusion process provided in an embodiment of the present invention;

[0059] Figure 9This is a schematic diagram of the process for determining the driver's state according to an embodiment of the present invention;

[0060] Figure 10 This is a schematic diagram of the process corresponding to controlling the autonomous driving of a vehicle, provided in an embodiment of the present invention;

[0061] Figure 11 This is a schematic diagram of the cabin supplemental lighting corresponding to a specific embodiment of the present invention;

[0062] Figure 12 This is a schematic diagram of image stitching and fusion corresponding to a specific embodiment of the present invention;

[0063] Figure 13 This is a schematic diagram of the driver status recognition device provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0065] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0066] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0067] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0068] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0069] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0070] Figure 1 This diagram illustrates the flow chart of the driver state recognition method provided in this embodiment of the invention; the executing entity may be a cockpit-driver integrated domain controller. Figure 2 This diagram illustrates the hardware structure of the integrated domain controller for the rack provided in an embodiment of the present invention.

[0071] Please see Figure 1 A method for driver state recognition, comprising:

[0072] Step S101: Perform real-time eye recognition on the current driver to obtain the eye region;

[0073] In one specific embodiment, real-time eye recognition of the current driver can be performed using millimeter-wave radar. Millimeter-wave radar emits 77GHz millimeter waves and can identify the distance and angle of objects in front, similar to lidar. After acquiring the distance and angle information, the millimeter-wave radar can virtually construct a 3D model based on point cloud data. From this 3D model, basic elements such as spheres and cylinders can be identified. For example, a human head is spherical, and the eyes and nose have specific features such as indentations and protrusions. Specifically, when the driver enters the cockpit, millimeter-wave radar is used to create a 3D facial model of the current driver, thereby identifying the driver's eye area.

[0074] Step S102: When the first light ray is projected onto the eye area, the eye area is image acquired by the first image acquisition device corresponding to the first light ray to obtain a black and white image; the first light ray matches the human eye's perception of light;

[0075] In one specific embodiment, in order to reduce the impact of the light source on the driver's vision, when supplementing the eye area with light, light that is not easily perceived by the human eye can be used; specifically, the first image acquisition device can be a camera, or a device with image acquisition function such as a smartphone or camera; the first light can be red light, and the first image acquisition device adapted to the first light can be a binocular infrared camera, and the images acquired by the infrared camera are all black and white images.

[0076] Step S103: When the second light beam is projected onto the remaining area of ​​the cockpit excluding the eye area, an image is acquired from the remaining area based on the second image acquisition device corresponding to the second light beam to obtain a color image;

[0077] In a specific embodiment, in order to ensure the image acquisition quality inside the cockpit, when supplementing the lighting of the remaining areas inside the cockpit other than the eye area, a second light source that can meet the brightness requirements of the scene can be used. Specifically, the second image acquisition device can be a camera, or a device with image acquisition function such as a smartphone or camera. The second light source can be white light. The second image acquisition device can be a binocular RGB camera (where R stands for red, G for green, and B for blue, and RGB is used to represent color mode). The RGB camera can acquire color images.

[0078] Step S104: Perform image fusion on the black and white image and the color image to obtain a fused image;

[0079] In one specific embodiment, the black and white image corresponding to the eye area and the color image corresponding to the remaining area are fused to obtain a fused image. The fused image includes all scene information inside the cockpit, such as the driver's face, driver's posture, and cockpit background.

[0080] Step S105: Determine the current driver's driving status based on the fused image.

[0081] In one specific embodiment, scene information in the fused image is analyzed to determine the driver's driving status, thereby avoiding accidents caused by poor driver condition and ensuring vehicle driving safety.

[0082] In one specific embodiment, the driver state recognition method described above can be implemented based on a chassis-integrated domain controller. (See also...) Figure 2The cockpit-driver integrated domain controller may include a fused SoC (System on Chip), an Ethernet switching unit, an AI (Artificial Intelligence) processing unit, an AI computing unit, a functional safety control unit, and a 5G / V2X (Vehicle to X) communication unit.

[0083] Specifically, the fused SoC integrates multiple computing units, primarily supporting intelligent cockpit-related functions, including in-vehicle occupant monitoring, human-machine interaction, and instrument display. The NPU (Network Processing Unit) integrated within the fused SoC carries AI vision algorithms such as DMS. The CPU (Central Processing Unit) integrated within the fused SoC processes the acquired millimeter-wave radar data to assist DMS in acquiring 3D human body information and posture information, aiding in the determination of the human eye region. The GPU (Graphics Processing Unit) integrated within the fused SoC can process 360° surround-view video signals for parking scenarios. For image data acquired by the first and second image acquisition devices, segmentation and stitching can be optionally achieved through the collaborative operation of the CPU and GPU units.

[0084] Visual signals exceeding the computing capabilities of the fused SoC are transmitted to the AI ​​processing module via the Ethernet bus set up in the Ethernet switching unit, where they are processed collaboratively using the module's larger internal computing power. The processed data is then returned to the fused SoC for further decision-making calculations. The AI ​​computing unit primarily processes driving-related visual algorithms and can also process some LiDAR algorithms, carrying visual computation data for forward, side, surround view, and parking AI algorithms.

[0085] The functional safety control unit is used to monitor the operating status of the cockpit-driver integrated domain controller. It also has some built-in driving algorithms, interfaces with the vehicle control execution unit, and issues vehicle control commands. The 5G / V2X module communicates with the cloud, roadside equipment, etc., and the relevant algorithm models and status monitoring are linked with the cloud platform.

[0086] In this embodiment of the invention, after determining the driver's eye area, a first light beam is projected onto the eye area, and a second light beam is projected onto the remaining area inside the cockpit excluding the eye area. By projecting different light beams to different areas to supplement the lighting inside the cockpit, the interference of the supplementary lighting source on the driver's vision can be reduced, thereby ensuring driving safety. At the same time, by dividing the area and projecting different light beams according to different areas, the image quality of the acquired images inside the cockpit can be guaranteed, thereby improving the accuracy of driver state analysis and recognition. Analyzing the driver's driving state based on the fused image can avoid dangers caused by poor driver condition, thereby ensuring the safety of vehicle operation.

[0087] Figure 3 This is a flowchart illustrating the process of determining the first projection attribute according to an embodiment of the present invention, as shown below. Figure 3 As shown, when the first light ray is projected onto the eye region, the process of acquiring an image of the eye region based on the first image acquisition device corresponding to the first light ray to obtain a black and white image includes:

[0088] Step S301: Based on the grid partitioning result, perform region detection within the cockpit to obtain a first grid partition corresponding to the eye region; the grid partitioning result is determined based on partitioning processing within the cockpit.

[0089] In a specific embodiment, the image acquisition area inside the cockpit is determined before grid partitioning; the image acquisition area is divided into multiple grids according to a preset grid size, thereby determining the corresponding positions of the eye area and the remaining area in the grid; the eye area is the first grid partition; the preset grid size can be determined according to the image acquisition area inside the cockpit, or the grid size can be fixed.

[0090] Step S302: Determine the first projection attribute of the light source projection point based on the first grid partition; the first projection attribute includes the target projection angle and the target projection direction;

[0091] In one specific embodiment, the projection attributes include projection angle and projection direction. The first projection attribute of the light source projection point is determined based on the position of the first grid partition to achieve accurate supplementary lighting for the first grid partition. Specifically, the light source projection point is located inside the cockpit and is capable of projecting at least two types of light, including a first light beam and a second light beam. Furthermore, the projection direction and angle of the light source projection point can be adjusted according to the driver's position.

[0092] Step S303: Based on the first projection angle, the first projection direction, and the first image acquisition device, an image of the eye region is acquired to obtain a black and white image.

[0093] In one specific embodiment, after determining the projection attributes of the light source projection point, the first grid partition, i.e. the eye region, is captured to obtain a black and white image including the eye region.

[0094] In this embodiment of the invention, adjusting the direction of the light source projection point according to the relative position of the eye region inside the cockpit before acquiring a black and white image can improve the accuracy of the first light projection.

[0095] Figure 4 This diagram illustrates the process of acquiring black and white images according to an embodiment of the present invention. Figure 4 As shown, the step of acquiring a black and white image of the eye region based on the first projection angle, the first projection direction, and the first image acquisition device includes:

[0096] Step S401: Based on the first projection angle and the first projection direction, project the first light beam onto the first grid partition;

[0097] In one specific embodiment, the light source projection point is adjusted to a position matching the first grid partition, i.e. the eye area, according to the first projection angle and the first projection direction corresponding to the first light ray, and the first light ray is projected to supplement the first grid partition with light.

[0098] Step S402: Based on the first image acquisition device, the first grid partition on which the first light beam has been projected is imaged to obtain the black and white image.

[0099] In one specific embodiment, the first image acquisition device acquires images of the first grid partition after supplemental lighting. The resulting eye image is clearer than the eye image acquired without supplemental lighting, which is beneficial for using AI algorithms to identify and analyze the driver's state.

[0100] In this embodiment of the invention, red light supplemental lighting is used for the eye area, which can reduce the interference of light on the driver's vision, thereby reducing the potential driving risks caused by poor driver vision and improving the safety of driving.

[0101] Figure 5 This diagram illustrates the process flow corresponding to the grid unit provided in an embodiment of the present invention, as follows: Figure 5 As shown, when the second light beam is projected onto the remaining area inside the cockpit excluding the eye area, the second image acquisition device corresponding to the second light beam acquires an image of the remaining area to obtain a color image, including:

[0102] Step S501: Based on the grid partitioning results, perform region detection within the cockpit to obtain a second grid partition corresponding to the remaining region;

[0103] In one specific embodiment, the relative position of the remaining area inside the cockpit is determined based on the grid partitioning results.

[0104] Step S502: Based on preset partitioning rules, the second grid partition is divided into grids to obtain multiple grid units; each grid unit includes multiple grids.

[0105] In one specific embodiment, since the remaining area includes the entire cockpit interior except for the eye area, to facilitate supplemental lighting in the remaining area, it is necessary to divide the multiple grids included in the second grid partition into multiple grid units, each comprising multiple grids. Depending on the area inside the cockpit, a grid unit may include at least one grid.

[0106] Step S503: Based on the second image acquisition device, image acquisition is performed on the plurality of grid units to obtain a color image.

[0107] In one specific embodiment, a second image acquisition device is used to sequentially acquire images of multiple grids to obtain a color image.

[0108] In this embodiment of the invention, dividing the remaining area with a large area into multiple grid units can reduce the number of times the projection attributes of the light source projection point are confirmed, save computing resources, and improve the convenience of light projection.

[0109] Figure 6 This is a schematic diagram of the process for determining the second projection attribute provided in an embodiment of the present invention, such as... Figure 6 As shown, the process of acquiring images from the plurality of grid units using the second image acquisition device to obtain a color image includes:

[0110] Step S601: Determine a second projection attribute of the light source projection point based on the plurality of grid units; the second projection attribute includes a plurality of projection angles corresponding to the plurality of grid units, and a plurality of projection directions corresponding to the plurality of grid units;

[0111] In one specific embodiment, the projection attributes of the light source projection point and each grid cell are determined according to the angle and direction of the multiple grid cells, so as to achieve accurate supplementary lighting for the multiple grid cells.

[0112] Step S602: Based on the multiple projection angles, the multiple projection directions, and the second image acquisition device, images are acquired from the multiple grid units to obtain color images.

[0113] In one specific embodiment, after determining multiple projection attributes of the light source projection point, multiple grid units, i.e. the remaining area, are collected to obtain a color image corresponding to the remaining area.

[0114] In this embodiment of the invention, adjusting the direction of the light source projection point according to the relative positions of multiple grid units inside the cockpit before color image acquisition can improve the accuracy of the second light projection.

[0115] Figure 7 This diagram illustrates the process of acquiring color images according to an embodiment of the present invention. Figure 7 As shown, the process of acquiring images from the multiple grid units based on the multiple projection angles, multiple projection directions, and the second image acquisition device to obtain a color image includes:

[0116] Step S701: Based on the projection angle and projection direction corresponding to each of the grid cells, project the second light beam sequentially onto each of the grid cells;

[0117] In one specific embodiment, according to each projection angle and each projection direction corresponding to each grid unit, the light source projection point is sequentially adjusted to a position matching each grid unit, and a second light beam is projected to supplement the lighting of multiple grid units, i.e., the second grid partition.

[0118] Step S702: Based on the second image acquisition device, image acquisition is performed on each of the grid units that have been projected with the second light, to obtain multiple images to be stitched together corresponding to each grid unit;

[0119] In one specific embodiment, the second image acquisition device acquires an image of each grid unit after supplemental lighting to obtain multiple corresponding images to be stitched together.

[0120] Step S703: Perform image stitching processing on the multiple images to be stitched to obtain the color image.

[0121] In one specific embodiment, multiple images to be stitched together are stitched together according to the relative positions of the grids to obtain a complete color image of the remaining area, which facilitates the use of AI algorithms to identify and analyze the driver's state.

[0122] In this embodiment of the invention, sequentially acquiring grid-by-grid images of multiple grid units after supplemental lighting and division can capture color images inside the cockpit, thereby improving the computational effect of the AI ​​algorithm. In addition, acquiring grid-by-grid images of multiple grid units can reduce the illuminance inside the cockpit per unit time, avoiding windshield reflections that may interfere with the driver's vision.

[0123] Figure 8 This is a schematic diagram of the image fusion process provided in the embodiments of the present invention, such as... Figure 8 As shown, the process of fusing the black-and-white image and the color image to obtain a fused image includes:

[0124] Step S801: Based on the black and white image and the color image, perform duplicate region detection to obtain the duplicate regions in the color image that are repeated with the black and white image;

[0125] In one specific embodiment, to ensure the fidelity of the fused image, it is necessary to perform duplicate region detection on the black-and-white image and the color image, and delete the regions in the color image that are duplicates of the black-and-white image.

[0126] Step S802: Delete the repeating regions in the color image to obtain the color image to be fused;

[0127] In one specific embodiment, after identifying the duplicate regions, the duplicate regions are removed from the color image to obtain the color image to be fused.

[0128] Step S803: Perform image fusion on the black and white image and the color image to be fused to obtain the fused image.

[0129] In one specific embodiment, the black and white image and the color image to be fused are fused according to the relative positions of the grid to obtain a fused image, which is the cockpit interior image after supplemental lighting, including the entire scene inside the cockpit.

[0130] In this embodiment of the invention, image fusion of the acquired black-and-white and color images can ensure the fidelity of the captured scene inside the cockpit, facilitating the identification and analysis of the driver's state.

[0131] Figure 9 This is a schematic diagram of the process for determining the driver's state according to an embodiment of the present invention, such as... Figure 9 As shown, determining the current driver's driving state based on the fused image includes:

[0132] Step S901: Based on the fused image, perform human posture detection and eye opening / closing detection on the current driver, and obtain posture detection results and eye opening / closing detection results respectively;

[0133] In one specific embodiment, the fused image includes all scene information inside the cockpit, such as the driver's face, driver's posture, and cockpit background; the posture detection results may include behaviors suspected of fatigued or dangerous driving, such as frequent forward leaning of the body, forward leaning of the driver's head, and the driver's face deviating from the driving direction for a long time; the opening and closing detection results are the degree of eye opening and closing.

[0134] Step S902: Based on the posture detection results and the opening / closing detection results, determine the current driving state of the driver.

[0135] In one specific embodiment, the current driver state is determined to be either a normal driving state or an abnormal driving state based on the posture detection results and / or opening / closing detection results.

[0136] In this embodiment of the invention, the driver's state is analyzed based on the acquired fused images, which can identify the driver's current driving state in real time and accurately, thereby ensuring driving safety.

[0137] Figure 10 This is a schematic diagram of the process corresponding to controlling the autonomous driving of a vehicle provided in an embodiment of the present invention, such as... Figure 10 As shown, after determining the current driver's driving state based on the opening / closing detection result, the method includes:

[0138] Step S1001: If the posture detection result indicates that the current driver's posture does not conform to the preset driving posture, or if the opening and closing detection result indicates that the current driver's eye opening and closing degree is lower than the preset opening and closing threshold, a driving takeover command is generated.

[0139] In one specific embodiment, when a person's face is detected to be deviating from the driving direction for an extended period of time, the opening of their eyes is below a threshold, or there are frequent behaviors such as the person or head tilting forward, which may indicate fatigued or dangerous driving, a driver takeover command is generated. The driver takeover command is used to control the vehicle to enter the autonomous driving mode, so as to take over the control of the vehicle from the driver.

[0140] Step S1002: Based on the driving takeover command, control the vehicle to enter the automatic driving mode.

[0141] In one specific embodiment, based on the driver takeover command and according to the vehicle's current driving status, the vehicle is controlled to enter an autonomous driving mode, which can perform actions such as deceleration and parking.

[0142] In this embodiment of the invention, when a driver is suspected of driving while fatigued or dangerously, the system can take over the vehicle being driven by the driver, thereby reducing the probability of safety accidents caused by fatigued or dangerous driving and ensuring driving safety.

[0143] Figure 11 This is a schematic diagram of the cabin supplemental lighting corresponding to a specific embodiment of the present invention; Figure 12This is a schematic diagram of image stitching and fusion corresponding to a specific embodiment of the present invention; in one specific embodiment, after dividing the cockpit interior into grid sections, corresponding light is sequentially projected onto each grid unit of the eye area and the remaining area, such as... Figure 11 As shown, when the area to be illuminated is the eye area, the first light ray is projected onto the first grid partition corresponding to the eye area; when the area to be illuminated is the remaining area, the projected light ray of the light source projection point is switched to the second light ray, and the second light ray is projected onto each grid unit in sequence.

[0144] Furthermore, after supplementing the lighting inside the cockpit, a first image acquisition device corresponding to the first ray is used to acquire images of the eye area, and a second image acquisition device corresponding to the second ray is used to acquire images of the remaining area; since the remaining area contains multiple grid units, multiple images to be stitched are acquired, such as... Figure 12 As shown, when acquiring images of the remaining area, the light source projection point projects a second ray onto each grid cell, that is, the second ray appears as a dynamically moving light spot in multiple grid cells of the remaining area; images are acquired for each grid cell through which the dynamically moving light spot passes, resulting in multiple frames of images to be stitched; the multiple frames of images to be stitched are stitched together per unit time to obtain a color image corresponding to the remaining area.

[0145] This invention also provides a device for driver status recognition, such as... Figure 13 As shown, the device includes:

[0146] The eye recognition module 1301 is used to perform real-time eye recognition on the current driver to obtain the eye area;

[0147] The black and white image acquisition module 1302, when a first ray is projected onto the eye region, acquires an image of the eye region based on a first image acquisition device corresponding to the first ray, and obtains a black and white image; the first ray matches the human eye's perception of light;

[0148] The color image acquisition module 1303 is used to acquire an image of the remaining area of ​​the cockpit, excluding the eye area, based on a second image acquisition device corresponding to the second light, when the second light is projected onto the remaining area of ​​the cockpit, and to obtain a color image.

[0149] Image fusion module 1304 is used to fuse the black and white image and the color image to obtain a fused image;

[0150] The state analysis module 1305 is used to determine the current driving state of the driver based on the fused image.

[0151] In other embodiments, the black-and-white image acquisition module 1302 further includes:

[0152] The first grid partitioning determination module is used to perform region detection within the cockpit based on the grid partitioning result to obtain a first grid partition corresponding to the eye region; the grid partitioning result is determined based on partitioning processing within the cockpit.

[0153] The first projection attribute determination module is used to determine the first projection attribute of the light source projection point based on the first grid partition; the first projection attribute includes the target projection angle and the target projection direction.

[0154] The first black-and-white image acquisition module is used to acquire images of the eye region based on the first projection angle, the first projection direction, and the first image acquisition device to obtain a black-and-white image.

[0155] In other embodiments, the first black-and-white image acquisition module further includes:

[0156] The first light projection module is used to project the first light into the first grid partition based on the first projection angle and the first projection direction;

[0157] The second black-and-white image acquisition module is used to acquire images of the first grid partition that has been projected with the first light, based on the first image acquisition device, to obtain the black-and-white image.

[0158] In other embodiments, the color image acquisition module 1303 further includes:

[0159] The second grid partitioning determination module is used to perform region detection in the cockpit based on the grid partitioning results, and obtain the second grid partitioning corresponding to the remaining region.

[0160] The grid unit division module is used to divide the second grid partition into multiple grid units based on a preset division rule; the grid unit includes multiple grids.

[0161] The first color image acquisition module is used to acquire images of the plurality of grid units based on the second image acquisition device to obtain a color image.

[0162] In other embodiments, the first color image acquisition module further includes:

[0163] The second projection attribute determination module is used to determine the second projection attribute of the light source projection point based on the plurality of grid units; the second projection attribute includes a plurality of projection angles corresponding to the plurality of grid units, and a plurality of projection directions corresponding to the plurality of grid units;

[0164] The second color image acquisition module is used to acquire images from the multiple grid units based on the multiple projection angles, the multiple projection directions, and the second image acquisition device, to obtain a color image.

[0165] In other embodiments, the second color image acquisition module further includes:

[0166] The second light projection module is used to project the second light beam sequentially onto each of the grid units based on the projection angle and projection direction corresponding to each of the grid units;

[0167] The third color image acquisition module is used to acquire images of each of the grid units that have been projected with the second light, based on the second image acquisition device, to obtain multiple images to be stitched together corresponding to each of the grid units;

[0168] The image stitching module is used to perform image stitching processing on the multiple images to be stitched together to obtain the color image.

[0169] In other embodiments, the image fusion module 1304 further includes:

[0170] The repeating region detection module is used to perform repeating region detection based on the black and white image and the color image to obtain repeating regions in the color image that are repeated with the black and white image;

[0171] The image deletion module is used to delete the repeating regions in the color image to obtain the color image to be fused.

[0172] The target image fusion module is used to perform image fusion on the black and white image and the color image to be fused to obtain the fused image.

[0173] In other embodiments, the state analysis module 1305 further includes:

[0174] The state detection module is used to perform human posture detection and eye opening and closing detection on the current driver based on the fused image, and obtain posture detection results and eye opening and closing detection results respectively;

[0175] The state determination module is used to determine the current driving state of the driver based on the attitude detection results and the opening / closing detection results.

[0176] In other embodiments, the device further includes:

[0177] The instruction generation module is used to generate a driving takeover instruction when the posture detection result indicates that the current driver's posture does not conform to the preset driving posture, or when the opening and closing detection result indicates that the current driver's eye opening and closing degree is lower than the preset opening and closing threshold.

[0178] The vehicle automatic control module is used to control the vehicle to enter the automatic driving mode based on the driver takeover command.

[0179] The apparatus and method embodiments described above are based on the same inventive concept and are used to implement the above-described driver state recognition method.

[0180] This invention also provides a driver state recognition device, the device comprising: a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the driver state recognition method as described in the method embodiment.

[0181] Embodiments of the present invention also provide a storage medium, which may be disposed in a server to store at least one instruction, at least one program, code set, or instruction set for implementing a data management method in the method embodiments, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the driver state recognition method as described in any of the method embodiments.

[0182] Optionally, in embodiments of the present invention, the storage medium may be located at at least one of a plurality of network servers in a computer network. Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0183] As can be seen from the embodiments provided by the present invention above, after determining the driver's eye area, the present invention projects a first light beam onto the eye area and a second light beam onto the remaining area inside the cockpit excluding the eye area. By projecting different light beams onto different areas to supplement the lighting inside the cockpit, the interference of the supplementary lighting source on the driver's vision can be reduced, thereby ensuring driving safety. At the same time, segmenting the area and projecting different light beams according to different areas can ensure the image quality of the acquired cockpit interior images, thereby improving the accuracy of driver state analysis and recognition. Analyzing the driver's driving state based on the fused image can avoid dangers caused by poor driver condition, thereby ensuring vehicle driving safety. Therefore, the present invention achieves the technical effect of improving the quality of acquired images while reducing interference with the driver's vision.

[0184] It should be noted that the various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for driver state recognition, characterized in that, The method includes: Real-time eye recognition is performed on the current driver to obtain the eye region; When a first ray of light is projected onto the eye region, an image of the eye region is acquired based on a first image acquisition device corresponding to the first ray of light, resulting in a black and white image; the first ray of light is matched with the human eye's perception of light; When the second light beam is projected onto the remaining area of ​​the cockpit excluding the eye area, an image is acquired from the remaining area based on the second image acquisition device corresponding to the second light beam to obtain a color image. The black-and-white image and the color image are fused to obtain a fused image; Based on the fused image, the current driver's posture detection and eye opening / closing detection are performed to obtain posture detection results and eye opening / closing detection results, respectively. The posture detection includes at least one of the following: human body leaning forward, head leaning forward, and face deviating from the driving direction. Based on the posture detection results and the opening / closing detection results, the current driving state of the driver is determined.

2. The method for driver state recognition according to claim 1, characterized in that, When the first light ray is projected onto the eye region, the process of acquiring an image of the eye region based on a first image acquisition device corresponding to the first light ray to obtain a black and white image includes: Based on the grid partitioning results, region detection is performed within the cockpit to obtain a first grid partition corresponding to the eye region; the grid partitioning results are determined based on the partitioning processing performed within the cockpit. The first projection attribute of the light source projection point is determined based on the first grid partition; the first projection attribute includes a first projection angle and a first projection direction. Based on the first projection angle, the first projection direction, and the first image acquisition device, an image of the eye region is acquired to obtain a black and white image.

3. The method for driver state recognition according to claim 2, characterized in that, The step of acquiring a black and white image of the eye region based on the first projection angle, the first projection direction, and the first image acquisition device includes: Based on the first projection angle and the first projection direction, the first light ray is projected onto the first grid partition; The first image acquisition device acquires an image of the first grid partition on which the first light has been projected, thereby obtaining the black and white image.

4. The method for driver state recognition according to claim 1, characterized in that, When the second light ray is projected onto the remaining area of ​​the cockpit excluding the eye region, the remaining area is image-captured by a second image acquisition device corresponding to the second light ray to obtain a color image, including: Based on the grid partitioning results, region detection is performed in the cockpit to obtain a second grid partitioning corresponding to the remaining region; Based on preset partitioning rules, the second grid partition is divided into multiple grid units; each grid unit includes multiple grids. The second image acquisition device acquires images from the multiple grid units to obtain a color image.

5. The method for driver state recognition according to claim 4, characterized in that, The step of acquiring images from the plurality of grid units using the second image acquisition device to obtain a color image includes: A second projection attribute is determined based on the plurality of grid cells; the second projection attribute includes a plurality of projection angles corresponding to the plurality of grid cells, and a plurality of projection directions corresponding to the plurality of grid cells; Based on the multiple projection angles, the multiple projection directions, and the second image acquisition device, images are acquired from the multiple grid units to obtain color images.

6. The method for driver state recognition according to claim 5, characterized in that, The step of acquiring images from the multiple grid units based on the multiple projection angles, multiple projection directions, and the second image acquisition device to obtain a color image includes: Based on the projection angle and projection direction corresponding to each of the grid cells, the second light ray is sequentially projected onto each of the grid cells; Based on the second image acquisition device, images are acquired for each of the grid units that have been projected with the second light, to obtain multiple images to be stitched together corresponding to each of the grid units; The multiple images to be stitched together are processed to obtain the color image.

7. The method for driver state recognition according to claim 1, characterized in that, The process of fusing the black-and-white image and the color image to obtain the fused image includes: Based on the black-and-white image and the color image, repeating region detection is performed to obtain repeating regions in the color image that are repeated in the black-and-white image; The repeated regions in the color image are deleted to obtain the color image to be fused; The black-and-white image and the color image to be fused are fused to obtain the fused image.

8. The method for driver state recognition according to claim 1, characterized in that, After determining the current driver's driving state based on the opening / closing detection result, the method includes: If the posture detection result indicates that the current driver's posture does not conform to the preset driving posture, or if the opening and closing detection result indicates that the current driver's eye opening and closing degree is lower than the preset opening and closing threshold, a driving takeover command is generated. Based on the aforementioned driver takeover command, the vehicle is controlled to enter autonomous driving mode.

9. A device for driver status recognition, characterized in that, The device includes: The eye recognition module is used to perform real-time eye recognition on the current driver to obtain the eye area; The black and white image acquisition module acquires an image of the eye region based on a first image acquisition device corresponding to the first light ray when the first light ray is projected onto the eye region, thereby obtaining a black and white image; the first light ray is matched with the human eye's perception of light; A color image acquisition module is used to acquire an image of the remaining area of ​​the cockpit, excluding the eye area, based on a second image acquisition device corresponding to the second light beam, to obtain a color image when the second light beam is projected onto the remaining area of ​​the cockpit, excluding the eye area. An image fusion module is used to fuse the black-and-white image and the color image to obtain a fused image; The state detection module is used to perform human posture detection and eye opening and closing detection on the current driver based on the fused image, and obtain posture detection results and eye opening and closing detection results respectively. The posture detection includes at least one of human body forward tilt, head forward tilt, and face deviation from the driving direction. The state determination module is used to determine the current driving state of the driver based on the attitude detection results and the opening / closing detection results.

10. An electronic device comprising a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the driver state recognition method as claimed in any one of claims 1-8.

11. A computer storage medium storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by a processor to implement the driver state recognition method as described in any one of claims 1-8.

Images