Driver concentration evaluation method, device, computer equipment and storage medium

Abstract

The present application relates to a method, device, computer equipment and storage medium for evaluating a driver's concentration, wherein the method comprises: detecting each frame of the driver's head and shoulder image, obtaining the driver's head posture angle data, and generating a head posture angle feature map corresponding to each head posture angle data; obtaining the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generating a vehicle posture angle feature map corresponding to each first posture angle data; extracting the driver's first binocular image from each frame of the head and shoulder image, and generating a gaze point feature map corresponding to each first binocular image; performing multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluating the driver's concentration based on the fusion result to obtain the driver's concentration level. Through the present application, the problem of being unable to accurately and comprehensively evaluate the driver's concentration is solved, and an accurate and comprehensive evaluation of the driver's concentration is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, computer equipment and storage medium for evaluating driver concentration. Background Art

[0002] During driving, the driver's concentration directly affects driving safety. Long-term driving, lack of sleep or physical discomfort often lead to driver inattention and reduce driving safety.

[0003] In order to solve the above problems, the existing driver status detection method needs to obtain multiple status information such as the driver's facial expression information, body movement information, and health index information, and comprehensively judge the driver's driving status. However, for ordinary vehicles, their internal hardware facilities cannot meet the collection requirements of the above methods, and thus cannot accurately and comprehensively evaluate the driver's attention.

[0004] There is currently no effective solution to the problem that related technologies cannot accurately and comprehensively assess driver attention. Summary of the invention

[0005] In this embodiment, a method, apparatus, computer device and storage medium for evaluating driver concentration are provided to solve the problem in the related art that it is impossible to accurately and comprehensively evaluate the driver's concentration.

[0006] In a first aspect, a method for evaluating a driver's concentration is provided in this embodiment, and the method includes:

[0007] Detecting each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generating a head posture angle feature map corresponding to each head posture angle data;

[0008] Acquire first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generate a vehicle posture angle feature map corresponding to each of the first posture angle data;

[0009] Extracting the first binocular image of the driver from each frame of the head-shoulder image, and generating a gaze point feature map corresponding to each of the first binocular images;

[0010] The head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are subjected to multimodal feature fusion, and the concentration of the driver is evaluated based on the fusion result to obtain the concentration level of the driver.

[0011] In some embodiments, detecting each frame of the driver's head and shoulder image to obtain the driver's head posture angle data includes:

[0012] Acquire multiple frames of the head and shoulder images of the driver;

[0013] By constraining the local neural domain model, the facial feature points in each frame of the head and shoulder image are detected to obtain the driver's head posture angle data.

[0014] In some embodiments, generating a head posture angle feature map corresponding to each head posture angle data includes:

[0015] Acquire the head pitch angle, head roll angle and head yaw angle in each of the head posture angle data;

[0016] Generate a first cumulative histogram corresponding to each of the head pitch angles, a second cumulative histogram corresponding to each of the head roll angles, and a third cumulative histogram corresponding to each of the head yaw angles;

[0017] The first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated to obtain the head posture angle feature map.

[0018] In some embodiments, generating a vehicle attitude angle characteristic graph corresponding to each of the first attitude angle data includes:

[0019] Obtaining a vehicle pitch angle, a vehicle roll angle, and a vehicle yaw angle in each of the first attitude angle data;

[0020] Generate a fourth cumulative histogram corresponding to each of the vehicle pitch angles, a fifth cumulative histogram corresponding to each of the vehicle roll angles, and a sixth cumulative histogram corresponding to each of the vehicle yaw angles;

[0021] The fourth cumulative histogram, the fifth cumulative histogram and the sixth cumulative histogram are integrated to obtain the vehicle attitude angle feature map.

[0022] In some embodiments, generating a gaze point feature map corresponding to each of the first binocular images includes:

[0023] Performing standardization processing on each of the first binocular images to obtain a corresponding second binocular image;

[0024] Processing each of the second binocular images by a convolutional neural network to obtain a corresponding first gaze vector;

[0025] Convert the first gaze vector from the world coordinate system to the camera coordinate system to obtain a second gaze vector;

[0026] A gaze point of the driver on the virtual visual field corresponding to each of the second gaze vectors is determined, and the gaze point feature map corresponding to each of the gaze points is generated.

[0027] In some embodiments, the step of normalizing each of the first binocular images to obtain a corresponding second binocular image includes:

[0028] Normalizing a first distance from a center point of both eyes to a center point of a camera in each of the first binocular images, and determining a corresponding scaling matrix according to the first standardized distance and the coordinates of the center point of both eyes;

[0029] Determine a camera transformation matrix based on a conversion process of the first binocular image from a camera coordinate system to a world coordinate system and the scaling matrix;

[0030] Determine a target conversion formula according to the camera conversion matrix and the real camera parameters and virtual camera parameters of the camera;

[0031] The first binocular image is transformed based on the target transformation formula to obtain the second binocular image.

[0032] In some embodiments, the multimodal feature fusion of the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map is performed, and the concentration of the driver is evaluated based on the fusion result to obtain the concentration level of the driver, including:

[0033] Based on the time dimension, the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are stacked to obtain a multimodal fusion feature map;

[0034] The multimodal fusion feature map is input into a concentration classification model for processing to obtain the concentration level of the driver.

[0035] In some embodiments, the step of inputting the multimodal fusion feature map into a concentration classification model for processing to obtain the concentration level of the driver includes:

[0036] One-dimensionalizing the multimodal fusion feature map;

[0037] Performing feature extraction on the one-dimensionalized multimodal fusion feature map to obtain a concentration feature value;

[0038] The concentration feature quantity is classified to obtain the concentration level of the driver.

[0039] In a second aspect, a driver's concentration evaluation device is provided in this embodiment, the device comprising: a first generation module, a second generation module, a third generation module and an evaluation module;

[0040] The first generating module is used to detect each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generate a head posture angle feature map corresponding to each head posture angle data;

[0041] The second generating module is used to obtain the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generate a vehicle posture angle feature map corresponding to each of the first posture angle data;

[0042] The third generating module is used to extract the first binocular image of the driver from each frame of the head and shoulder image, and generate a gaze point feature map corresponding to each of the first binocular images;

[0043] The evaluation module is used to perform multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluate the driver's concentration based on the fusion result to obtain the driver's concentration level.

[0044] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the driver concentration assessment method described in the first aspect when executing the computer program.

[0045] In a fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for evaluating the driver's concentration described in the first aspect is implemented.

[0046] Compared with the related art, the driver's concentration evaluation method, device, computer equipment and storage medium provided in this embodiment obtain the driver's head posture angle data by detecting each frame of the driver's head and shoulder image, and generate a head posture angle feature map corresponding to each head posture angle data; obtain the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generate a vehicle posture angle feature map corresponding to each first posture angle data; extract the driver's first binocular image from each frame of the head and shoulder image, and generate a gaze point feature map corresponding to each first binocular image; perform multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, evaluate the driver's concentration based on the fusion result, and obtain the driver's concentration level, which solves the problem of being unable to accurately and comprehensively evaluate the driver's attention, realizes accurate and comprehensive evaluation of the driver's attention, and reduces the hardware equipment requirements for the actual vehicle.

[0047] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 This is a hardware structure block diagram of a terminal device of a driver concentration evaluation method provided by an embodiment of the present application;

[0050] Figure 2 is a flow chart of a method for evaluating driver concentration provided by an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of a binocular image provided by an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of head posture angle data provided by an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of calculating gaze points based on binocular images provided by an embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of the process of feature graph classification processing provided by an embodiment of the present application;

[0055] Figure 7 It is a flowchart of a method for evaluating a driver's concentration provided by an embodiment of the present application;

[0056] Figure 8 is a flow chart of a method for evaluating driver concentration provided by a preferred embodiment of the present application;

[0057] Fig. 9 It is a structural block diagram of a driver concentration assessment device provided in one embodiment of the present application.

[0058] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, first generation module; 20, second generation module; 30, third generation module; 40, evaluation module. DETAILED DESCRIPTION

[0059] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0060] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0061] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the driver's concentration evaluation method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The terminal may further include a transmission device 106 and an input / output device 108 for communication functions. A person skilled in the art may understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0062] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the driver's concentration evaluation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0063] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0064] In this embodiment, a method for evaluating a driver's concentration is provided. Figure 2 is a flow chart of the driver's concentration evaluation method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0065] Step S210, detecting each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generating a head posture angle feature map corresponding to each head posture angle data;

[0066] Step S220, obtaining first attitude angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generating a vehicle attitude angle feature map corresponding to each first attitude angle data;

[0067] Step S230, extracting the first binocular image of the driver from each frame of the head and shoulder image, and generating a gaze point feature map corresponding to each first binocular image;

[0068] Step S240, performing multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluating the driver's concentration based on the fusion result to obtain the driver's concentration level.

[0069] Specifically, a camera device installed at a preset position, such as a camera device set in front of the driver's seat, is used to collect multiple frames of head and shoulder images of the driver, and each frame of the collected head and shoulder images is detected to obtain head posture angle data corresponding to each frame of the head and shoulder images, and the head posture angle data includes the driver's head pitch angle, head roll angle and head yaw angle in the head and shoulder images, wherein the head pitch angle refers to the angle of the head tilting forward and backward, the head roll angle refers to the angle of the head tilting left and right, and the head yaw angle refers to the rotation angle of the head in the horizontal direction. A first cumulative histogram corresponding to each head pitch angle, a second cumulative histogram corresponding to each head roll angle, and a third cumulative histogram corresponding to each head yaw angle are generated, and the first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated and processed to obtain a head posture angle feature map.

[0070] While collecting the driver's head and shoulder image, the first attitude angle data of the current vehicle is collected through sensors such as a gyroscope, wherein the vehicle types include automobiles, ships, etc. The vehicle pitch angle, vehicle roll angle, and vehicle yaw angle in each first attitude angle data are obtained, wherein the vehicle pitch angle refers to the angle at which the vehicle tilts forward and backward, the vehicle roll angle refers to the angle at which the vehicle tilts left and right, and the vehicle yaw angle refers to the horizontal rotation angle of the vehicle. A fourth cumulative histogram corresponding to each vehicle pitch angle, a fifth cumulative histogram corresponding to each vehicle roll angle, and a sixth cumulative histogram corresponding to each vehicle yaw angle are generated, and the fourth cumulative histogram, the fifth cumulative histogram, and the sixth cumulative histogram are integrated and processed to obtain a vehicle attitude angle feature map.

[0071] Furthermore, each frame of the head and shoulder image is processed by the face detection algorithm and the key point positioning algorithm to locate the driver's head position, such as Figure 3 As shown, the key point detection is combined to locate the position of both eyes, and then the first binocular image of the driver is extracted from each frame of the head and shoulder image. The binocular image only contains the driver's eyes. For example, the facial coordinates in each frame of the head and shoulder image are obtained by constraining the local neural domain model, and the first binocular image is cropped according to the binocular area. Alternatively, the neural network model is trained to directly extract the first binocular image of the driver from the head and shoulder image, which can be implemented by an end-to-end convolutional neural network model. This embodiment does not limit the method for obtaining the binocular image.

[0072] The first distance from the center point of both eyes to the center point of the camera in each first binocular image is standardized, and the corresponding scaling matrix is ​​determined according to the standardized first distance and the coordinates of the center point of both eyes; the camera transformation matrix is ​​determined based on the conversion process of the first binocular image from the camera coordinate system to the world coordinate system and the scaling matrix, and the target transformation formula is determined according to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera, and the first binocular image is transformed based on the target transformation formula to obtain a second binocular image; each second binocular image is processed by a convolutional neural network to obtain a corresponding first gaze vector, and the first gaze vector is converted from the world coordinate system to the camera coordinate system to obtain a second gaze vector, so as to determine the gaze point of the driver corresponding to each second gaze vector on the virtual field of view, and generate a gaze point feature map corresponding to each gaze point.

[0073] Afterwards, the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are subjected to multimodal feature fusion to obtain the corresponding multimodal fusion feature map, and the multimodal fusion feature map is classified to obtain the driver's concentration level. Among them, the multimodal fusion feature map can be input into the concentration classification model for processing to obtain the driver's concentration level; or, the currently acquired multimodal fusion feature map is matched with the preset feature map template to determine the feature map template with higher similarity, and the concentration level corresponding to the feature map template is used as the current driver's concentration level; or, the multimodal fusion feature map is subjected to dimensionality reduction processing, and the distribution relationship between data points is observed in a visual way, thereby realizing the classification judgment of the multimodal fusion feature map and obtaining the driver's concentration level.

[0074] The above-mentioned concentration level is used to reflect the driver's concentration level or whether he is in a state of fatigue. As the concentration level value gradually increases, the driver's concentration corresponding to the level value can be set to change from low to high or from high to low. For example, when the concentration level value is larger and the driver's concentration is lower, the output concentration level is level one, which indicates that the driver's concentration is high, reflecting that the driver's attention is relatively concentrated or not in a state of fatigue; the output concentration level is level five, which indicates that the driver's concentration is low, reflecting that the driver is not focused or is in a state of fatigue. When it is detected that the driver is not focused or is in a state of fatigue, the driver can be warned by means of sound alarms, vibration seats, etc.; or, the automatic driving function is activated, and the system temporarily takes over the vehicle control to ensure the current vehicle's driving safety; or, the driver's fatigue or inattention is recorded, and data analysis is performed to facilitate subsequent improvements and interventions by the driver and relevant managers.

[0075] It should be noted that different head posture angle data indicate different head postures of the driver, such as forward leaning, backward leaning or deflection, etc. The change of the driver's attention distribution is usually accompanied by continuous changes in head posture and eye direction. Different driver head posture angle data and eye directions reflect different behavioral characteristics and attention levels of the driver. In addition, in addition to using the cumulative histogram method, heat maps or other forms of feature maps can also be used to generate head posture angle data, vehicle feature angle data, and feature maps corresponding to gaze points.

[0076] In the existing driver status detection method, it is necessary to obtain multiple status information such as the driver's facial expression information, body movement information, and health index information, and comprehensively judge the driver's driving status. However, for ordinary vehicles, their internal hardware facilities cannot meet the collection requirements of the above methods, and thus cannot accurately and comprehensively evaluate the driver's attention.

[0077] Compared with the prior art, the present application detects each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generates a head posture angle feature map corresponding to each head posture angle data; obtains the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generates a vehicle posture angle feature map corresponding to each first posture angle data; extracts the driver's first binocular image from each frame of the head and shoulder image, and generates a gaze point feature map corresponding to each first binocular image; performs multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map, and the gaze point feature map, and evaluates the driver's concentration based on the fusion result to obtain the driver's concentration level. Based on this, the head posture angle feature map, vehicle posture angle feature map and gaze point feature map are subjected to multimodal feature fusion, and the driver's concentration assessment result is obtained by classifying the fused features. In this way, accurate concentration assessment can be achieved without using facial expression information, health indicator information and other status information that is difficult to collect, thereby solving the problem of being unable to accurately and comprehensively assess the driver's attention. Accurate and comprehensive assessment of the driver's attention is achieved. At the same time, there is no need to add high-cost information collection equipment inside the vehicle, which reduces the hardware equipment requirements for the actual vehicle.

[0078] In some embodiments, the step S210 of detecting each frame of the driver's head and shoulder image to obtain the driver's head posture angle data includes the following steps:

[0079] Acquire multiple frames of head and shoulder images of the driver;

[0080] By constraining the local neural domain model, the facial feature points in each frame of the head and shoulder image are detected to obtain the driver's head posture angle data.

[0081] Specifically, the camera device is installed at a preset position inside the vehicle to collect the driver's head and shoulder image, preferably in front of the driving position. Each frame of the collected head and shoulder image is detected to obtain the head posture angle data corresponding to each frame of the head and shoulder image. The head posture angle data includes the driver's head pitch angle, head roll angle and head yaw angle in the head and shoulder image. The head pitch angle refers to the angle of the head tilting forward and backward, the head roll angle refers to the angle of the head tilting left and right, and the head yaw angle refers to the rotation angle of the head in the horizontal direction.

[0082] Among them, the constrained local neural field (CLNF) model is used to detect and track the facial key points of each frame of the head and shoulder image. The model initializes the face position in each frame of the head and shoulder image, obtains the facial feature points of each frame, and searches and matches the feature points in its neighborhood position to achieve the detection and tracking of facial feature points. Figure 4 As shown, the constrained local neural domain uses the three-dimensional representation of facial landmarks internally, that is, by modeling and describing the three-dimensional positions of facial feature points to capture their spatial structure and morphological characteristics, head posture angle data can be extracted through camera projection. It should be noted that the above process mainly includes two parts: model construction and point fitting, and model construction includes point distribution model (PDM) construction and detailed modeling of local areas (Patch).

[0083] Through this embodiment, multiple frames of head and shoulder images of the driver are obtained, and the facial feature points in each frame of the head and shoulder image are detected by constraining the local neural domain model to accurately obtain the driver's head posture angle data.

[0084] In some embodiments, generating a head posture angle feature map corresponding to each head posture angle data in step S210 includes the following steps:

[0085] Obtaining the head pitch angle, head roll angle and head yaw angle in each head posture angle data;

[0086] Generate a first cumulative histogram corresponding to each head pitch angle, a second cumulative histogram corresponding to each head roll angle, and a third cumulative histogram corresponding to each head yaw angle;

[0087] The first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated to obtain a head posture angle feature map.

[0088] Specifically, the head pitch angle, head roll angle and head yaw angle in each head posture angle data are obtained, and the first cumulative histogram corresponding to each head pitch angle, the second cumulative histogram corresponding to each head roll angle and the third cumulative histogram corresponding to each head yaw angle are generated, and the first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated and processed to obtain the head posture angle feature map. Among them, the collected n frames of head and shoulder images are divided into multiple volumes, each volume of head and shoulder images contains m frames of head and shoulder images, and the head posture angle feature map corresponding to the i-th volume of head and shoulder images is as follows:

[0089]

[0090] In formula (1), P i Y represents the cumulative histogram of the head pitch angle corresponding to the i-th volume head and shoulder image; i R represents the cumulative histogram of the head yaw angle corresponding to the i-th volume head and shoulder image; i represents the cumulative histogram of the head roll angle corresponding to the i-th volume head and shoulder image; p j1 Indicates the predicted pitch angle a1≤-b1 corresponding to the jth volume, where b1 is the preset threshold; p j2 Indicates the predicted pitch angle a1<b1 in the jth volume, and -b1<a1; p j3 Indicates the predicted pitch angle a1≥b1 corresponding to the jth volume; y j1 Indicates the predicted yaw angle a2≤-b2 corresponding to the jth volume, where b2 is the preset threshold; y j2 Indicates that the predicted yaw angle in the jth volume is a2<b2, and -b2<a2; y j3 Indicates the predicted yaw angle a2≥b2 corresponding to the jth volume; r j1 Indicates the predicted roll angle a3≤-b3 corresponding to the jth volume, where b3 is the preset threshold; r j2 Indicates that the predicted roll angle in the jth volume is a3<b3, and -b3<a3; r j3 Indicates that the predicted roll angle a3 ≥ b3 in the jth volume. Based on this, the head posture angle feature map corresponding to the n-frame head and shoulder image is as follows:

[0091]

[0092] In formula (2), P H represents the first cumulative histogram corresponding to the n-frame head-shoulder image; P i Y represents the cumulative histogram of the head pitch angle corresponding to the i-th volume head and shoulder image; H Y represents the third cumulative histogram corresponding to the n-frame head-shoulder image; i R represents the cumulative histogram of the head yaw angle corresponding to the i-th volume head and shoulder image;H R represents the second cumulative histogram corresponding to the n-frame head-shoulder image; i Represents the cumulative histogram of the head roll angle corresponding to the i-th volume of head and shoulder image.

[0093] Through this embodiment, the head pitch angle, head roll angle and head yaw angle in each head posture angle data are obtained, and a first cumulative histogram corresponding to each head pitch angle, a second cumulative histogram corresponding to each head roll angle and a third cumulative histogram corresponding to each head yaw angle are generated, so that the first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated and processed to obtain a head posture angle feature map, thereby generating an accurate head posture angle feature map, which helps to improve the accuracy of subsequent multimodal feature fusion results.

[0094] In some embodiments, the step S220 of generating a vehicle attitude angle characteristic graph corresponding to each first attitude angle data includes the following steps:

[0095] Step S221, obtaining the vehicle pitch angle, vehicle roll angle and vehicle yaw angle in each first attitude angle data;

[0096] Step S222, generating a fourth cumulative histogram corresponding to the pitch angles of each vehicle, a fifth cumulative histogram corresponding to the roll angles of each vehicle, and a sixth cumulative histogram corresponding to the yaw angles of each vehicle;

[0097] Step S223, integrating the fourth cumulative histogram, the fifth cumulative histogram and the sixth cumulative histogram to obtain a vehicle attitude angle feature map.

[0098] Specifically, while collecting the driver's head and shoulder image, the first attitude angle data of the front vehicle is collected through sensors such as a gyroscope, and each first attitude angle data obtained includes the vehicle pitch angle, the vehicle roll angle and the vehicle yaw angle, wherein the vehicle pitch angle refers to the angle of the vehicle's front and rear tilt, the vehicle roll angle refers to the angle of the vehicle's left and right tilt, and the vehicle yaw angle refers to the vehicle's horizontal rotation angle.

[0099] Furthermore, a fourth cumulative histogram corresponding to the pitch angle of each vehicle, a fifth cumulative histogram corresponding to the roll angle of each vehicle, and a sixth cumulative histogram corresponding to the yaw angle of each vehicle are generated, and the fourth cumulative histogram, the fifth cumulative histogram, and the sixth cumulative histogram are integrated to obtain a vehicle attitude angle feature map. The specific expression is as follows:

[0100]

[0101] In formula (3), P V represents the fourth cumulative histogram corresponding to the n-frame head-shoulder image; P viY represents the cumulative histogram of the vehicle pitch angle corresponding to the i-th volume head and shoulder image; V Y represents the sixth cumulative histogram corresponding to the n-frame head-shoulder image; vi R represents the cumulative histogram of the vehicle yaw angle corresponding to the head-shoulder image of the i-th volume; V R represents the fifth cumulative histogram corresponding to the n-frame head-shoulder image; vi It represents the cumulative histogram of the vehicle roll angle corresponding to the i-th head-shoulder image.

[0102] Through this embodiment, the vehicle pitch angle, vehicle roll angle and vehicle yaw angle in each first attitude angle data are obtained, and a fourth cumulative histogram corresponding to each vehicle pitch angle, a fifth cumulative histogram corresponding to each vehicle roll angle and a sixth cumulative histogram corresponding to each vehicle yaw angle are generated. The fourth cumulative histogram, the fifth cumulative histogram and the sixth cumulative histogram are integrated and processed to generate an accurate vehicle attitude angle feature map, which helps to improve the accuracy of subsequent multimodal feature fusion results.

[0103] In some embodiments, generating a gaze point feature map corresponding to each first binocular image in step S230 includes the following steps:

[0104] Step S231, performing standardization processing on each first binocular image to obtain a corresponding second binocular image;

[0105] Step S232, processing each second binocular image by a convolutional neural network to obtain a corresponding first gaze vector;

[0106] Step S233, converting the first gaze vector from the world coordinate system to the camera coordinate system to obtain a second gaze vector;

[0107] Step S234, determining the gaze point of the driver on the virtual field of view corresponding to each second gaze vector, and generating a gaze point feature map corresponding to each gaze point.

[0108] Specifically, the first distance from the center point of both eyes to the center point of the camera in each first binocular image is standardized, and the corresponding scaling matrix is ​​determined based on the standardized first distance and the coordinates of the center point of both eyes, and the camera transformation matrix is ​​determined based on the conversion process of the first binocular image from the camera coordinate system to the world coordinate system, and the scaling matrix. According to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera, the target transformation formula is determined, and then the first binocular image is transformed based on the target transformation formula to obtain the second binocular image. This standardization process ensures the scale consistency of the binocular image, so that the image captured by the camera device is accurately mapped to the world coordinate system, so as to facilitate subsequent accurate spatial analysis.

[0109] Each second binocular image is processed by a convolutional neural network (CNN) to obtain a first gaze vector in a standardized space. The convolutional neural network used includes but is not limited to a residual network (ResNet), AlexNet, Inception series networks, VGG series networks, and other networks improved based on such networks.

[0110] The camera transformation matrix is ​​used to transform the first gaze vector from the world coordinate system to the camera coordinate system to obtain the second gaze vector. The specific formula is as follows:

[0111]

[0112] In formula (4), g = (g x ,g y ,g z ) represents the second gaze vector; M represents the camera transformation matrix; g` represents the first gaze vector.

[0113] Furthermore, the gaze point of the driver on the virtual horizon is determined according to the gaze vector in the camera coordinate system. Figure 5 As shown, h is the preset virtual horizon, which refers to the plane perpendicular to the driving direction of the vehicle, that is, the plane perpendicular to the driver. Figure 5 middle The expression is as follows:

[0114]

[0115] In formula (5), point A is the center point of both eyes, and the coordinates of point A in the camera coordinate system are (h x ,h y ,h z ); Point B is the gaze point on the virtual horizon; g=(g x ,g y ,g z ) represents the second gaze vector. The camera angle is α, that is, the angle between the Z axis and the Z' axis. The origin of the camera coordinate system is O. Let point C be equal to the origin of the camera coordinate system O, and point B' be the intersection of the vertical line passing through the origin O and AB. Then the angle β between OB and OB' is 0, that is, ,at this time The specific formula is as follows:

[0116]

[0117] Since point B' is on the Z axis, h z +mg z =0, then according to formula (6) we can get m=-h z / gz , substitute this formula into the above formula (5) to calculate the gaze point B'(x,y), which is the gaze point corresponding to the second gaze vector. Then, the gaze point corresponding to each second gaze vector is processed using the cumulative histogram to obtain the gaze point feature map C = (C1, C2, ..., C i ,…).

[0118] It should be noted that after obtaining multiple frames of binocular images of the driver, the pixel displacement information in each binocular image can also be used to track the position changes of the same pixel points in different image frames to calculate the driver's line of sight direction and the position of the gaze point, and generate a corresponding gaze point feature map based on each gaze point.

[0119] Through this embodiment, each first binocular image is standardized to obtain a corresponding second binocular image, each second binocular image is processed by a convolutional neural network to obtain a corresponding first gaze vector, and the first gaze vector is converted from the world coordinate system to the camera coordinate system to obtain a second gaze vector, and the gaze point of the driver corresponding to each second gaze vector on the virtual field of view is determined, and a gaze point feature map corresponding to each gaze point is generated to generate an accurate gaze point feature map, which helps to improve the accuracy of subsequent multimodal feature fusion results.

[0120] In some embodiments, the step S231 of normalizing each first binocular image to obtain a corresponding second binocular image includes the following steps:

[0121] Normalizing a first distance from a center point of both eyes to a center point of a camera in each first binocular image, and determining a corresponding scaling matrix according to the standardized first distance and the coordinates of the center point of both eyes;

[0122] Determine a camera transformation matrix based on a conversion process of the first binocular image from a camera coordinate system to a world coordinate system and a scaling matrix;

[0123] Determine the target transformation formula according to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera;

[0124] The first binocular image is transformed based on the target transformation formula to obtain a second binocular image.

[0125] Specifically, the first binocular image is converted from the camera coordinate system to the world coordinate system. The camera coordinate system is rotated so that the Z axis of the camera coordinate system points to the center of the two eyes, and then the camera coordinate system is rotated around the Z axis until the X axis of the camera coordinate system is coplanar with the head coordinate system. This conversion process is recorded as R c .

[0126] Furthermore, the Z axis of the camera coordinate system is scaled so that the distance from the center of both eyes to the center of the camera is a fixed value d, that is, the distance from the center of both eyes to the origin of the camera coordinate system is a fixed value d, so as to standardize the distance from the center of both eyes to the center of the camera. The specific formula of the scaling matrix S is as follows:

[0127]

[0128] In formula (7), h is the three-dimensional coordinate of the binocular center point in the camera coordinate system. The conversion process R from the camera coordinate system to the world coordinate system based on the first binocular image is: c And the scaling matrix S, calculate the camera transformation matrix, the specific formula is as follows:

[0129]

[0130] In formula (8), M represents the camera transformation matrix; S represents the scaling matrix; R c Represents the conversion process of the first binocular image from the camera coordinate system to the world coordinate system.

[0131] Afterwards, the target transformation formula is determined according to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera. The specific formula is as follows:

[0132]

[0133] In formula (9), W represents the target transformation formula; C s represents the real camera parameters; C r represents the virtual camera parameters; M represents the camera transformation matrix. The first binocular image is transformed using the target transformation formula to obtain a second binocular image, i.e., a standardized binocular image.

[0134] Through this embodiment, the first distance from the center point of both eyes to the center point of the camera in each first binocular image is standardized, and the corresponding scaling matrix is ​​determined based on the standardized first distance and the coordinates of the center point of both eyes. Based on the conversion process of the first binocular image from the camera coordinate system to the world coordinate system and the scaling matrix, the camera transformation matrix is ​​determined. According to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera, the target transformation formula is determined, and then the first binocular image is transformed based on the target transformation formula to obtain the second binocular image, so as to convert the original binocular image into a standardized binocular image, thereby improving the accuracy of the subsequent acquisition of the gaze vector in the standardized space.

[0135] In some embodiments, the step S240 of performing multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluating the driver's concentration based on the fusion result to obtain the driver's concentration level includes the following steps:

[0136] Step S241, based on the time dimension, stacking the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map to obtain a multimodal fusion feature map;

[0137] Step S242: input the multimodal fusion feature map into a concentration classification model for processing to obtain the driver's concentration level.

[0138] Specifically, according to the time dimension, the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are stacked to obtain a multimodal fusion feature map. The specific expression is as follows:

[0139]

[0140] In formula (10), f represents the multimodal fusion feature map obtained by stacking according to the time dimension; C represents the gaze point feature map; P H , Y H , R H represents the head posture angle feature map; P V , Y V , R V Represents the vehicle attitude angle feature map.

[0141] Furthermore, the multimodal fusion feature map is input into the concentration classification model for classification processing to obtain the driver's concentration level. As the concentration level value gradually increases, the driver's concentration corresponding to the level value can be set to change from low to high or from high to low. For example, when the concentration level value is larger and the driver's concentration is lower, the output concentration level is level one, indicating that the driver's concentration is high, reflecting that the driver's attention is relatively focused or not in a state of fatigue; the output concentration level is level five, indicating that the driver's concentration is low, reflecting that the driver is not focused or is in a state of fatigue.

[0142] It should be noted that when the system detects that the driver is inattentive or fatigued, the system may use methods such as sound alarms and seat vibration to warn the driver; or, the automatic driving function may be activated, and the system may temporarily take over the control of the vehicle to ensure the current driving safety of the vehicle; or, the driver's fatigue or inattention may be recorded, and data analysis may be performed to enable the driver and relevant managers to make subsequent improvements and interventions.

[0143] In addition to using the concentration classification model to classify the concentration level, the currently acquired multimodal fusion feature map can also be matched with the preset feature map template to determine the feature map template with higher similarity, and the concentration level corresponding to the feature map template can be used as the current driver's concentration level, or the multimodal fusion feature map can be subjected to dimensionality reduction processing, and the distribution relationship between data points can be observed in a visual way, thereby realizing the classification judgment of the multimodal fusion feature map and obtaining the driver's concentration level.

[0144] Through this embodiment, based on the time dimension, the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are stacked to obtain a multimodal fusion feature map, and the multimodal fusion feature map is input into the concentration classification model for processing to obtain the driver's concentration level. In this way, multiple feature items are combined as the basis for concentration evaluation to ensure the accuracy and comprehensiveness of the evaluation.

[0145] In some embodiments, the step S242 of inputting the multimodal fusion feature map into the concentration classification model for processing to obtain the driver's concentration level includes the following steps:

[0146] Convert the multimodal fusion feature map into one dimension;

[0147] Perform feature extraction on the one-dimensional multimodal fusion feature map to obtain the concentration feature value;

[0148] The concentration feature quantities are classified to obtain the driver's concentration level.

[0149] Specifically, Figure 6 As shown in the figure, the multimodal fusion feature map is input into the concentration classification model, the multimodal fusion feature map is flattened by the flatten layer in the model, and the one-dimensionalized multimodal fusion feature map is extracted by the long short-term memory (LSTM) layer in the model to obtain the concentration feature quantity, and the concentration feature quantity is input into the classification layer of the model, the concentration feature quantity is classified, and the driver's concentration level is obtained.

[0150] Among them, LSTM layers include but are not limited to long short-term memory neural networks, convolutional long short-term memory networks (ConvLSTM), bidirectional convolutional long short-term memory networks (bidirectional ConvLSTM) and LSTM-like networks; classification layers include but are not limited to a combination of fully connected layers (FC) and softmax classification layers, fully convolutional neural networks (FCN) and support vector machines (SVM).

[0151] Through this embodiment, the multimodal fusion feature map is converted into one dimension, and feature extraction is performed on the one-dimensional multimodal fusion feature map to obtain a concentration feature value, and then the multimodal fusion feature map is classified to accurately obtain the driver's concentration level.

[0152] The present embodiment is described and illustrated by means of specific examples below.

[0153] Figure 7 is a flow chart of the driver's concentration evaluation method of this embodiment, such as Figure 7 As shown, the driver's concentration evaluation method specifically includes the following steps:

[0154] Collect multiple frames of head and shoulder images of the driver S701, perform facial key point detection and tracking on each frame of the collected head and shoulder images S702, obtain head posture angle data corresponding to each frame of the head and shoulder images S703, generate head posture angle feature maps corresponding to each head posture angle data, and extract the first binocular image of the driver from each frame of the head and shoulder images S704, perform standardization on each first binocular image to obtain the corresponding second binocular image S705, process each second binocular image through a convolutional neural network to obtain the corresponding first gaze vector, convert the first gaze vector from the world coordinate system to the camera coordinate system to obtain the second gaze vector S706, and then determine the gaze point of the driver on the virtual horizon corresponding to each second gaze vector, and generate the gaze point feature map corresponding to each gaze point. While collecting the head and shoulder images, obtain the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder images S707, and generate the vehicle posture angle feature map corresponding to each first posture angle data.

[0155] Furthermore, based on the time dimension, the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are stacked to obtain a multimodal fusion feature map S708, and the multimodal fusion feature map is input into the concentration classification model for classification processing to obtain the driver's concentration level S709.

[0156] The present embodiment is described and illustrated below through preferred embodiments.

[0157] Figure 8 is a flow chart of the driver's concentration evaluation method of this preferred embodiment, such as Figure 8 As shown, the driver's concentration evaluation method includes the following steps:

[0158] Step S810, obtaining multiple frames of head and shoulder images of the driver, detecting facial feature points in each frame of the head and shoulder image by using a constrained local neural domain model, and obtaining head posture angle data of the driver;

[0159] Step S820, obtaining the head pitch angle, head roll angle and head yaw angle in each head posture angle data;

[0160] Step S830, generating a first cumulative histogram corresponding to each head pitch angle, a second cumulative histogram corresponding to each head roll angle, and a third cumulative histogram corresponding to each head yaw angle;

[0161] Step S840, integrating the first cumulative histogram, the second cumulative histogram and the third cumulative histogram to obtain a head posture angle feature map;

[0162] Step S850, obtaining first attitude angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generating a vehicle attitude angle feature map corresponding to each first attitude angle data;

[0163] Step S860, extracting the first binocular image of the driver from each frame of the head-shoulder image, and generating a gaze point feature map corresponding to each first binocular image;

[0164] Step S870, based on the time dimension, stacking the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map to obtain a multimodal fusion feature map;

[0165] Step S880: input the multimodal fusion feature map into the concentration classification model for processing to obtain the driver's concentration level.

[0166] Through this embodiment, multiple frames of head and shoulder images of the driver are obtained, and the facial feature points in each frame of the head and shoulder image are detected by a constrained local neural domain model to obtain the driver's head posture angle data, and the head pitch angle, head roll angle and head yaw angle in each head posture angle data are obtained, and a first cumulative histogram corresponding to each head pitch angle, a second cumulative histogram corresponding to each head roll angle and a third cumulative histogram corresponding to each head yaw angle are generated, and the first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated and processed to obtain a head posture angle feature map.

[0167] At the same time, the first attitude angle data of the current vehicle corresponding to each frame of the head and shoulder image is obtained, and the vehicle attitude angle feature map corresponding to each first attitude angle data is generated, and the first binocular image of the driver is extracted from each frame of the head and shoulder image to generate the gaze point feature map corresponding to each first binocular image. Based on the time dimension, the head attitude angle feature map, the vehicle attitude angle feature map and the gaze point feature map are stacked to obtain a multimodal fusion feature map, and then the multimodal fusion feature map is input into the concentration classification model for processing to obtain the driver's concentration level, which solves the problem of being unable to accurately and comprehensively evaluate the driver's attention, realizes accurate and comprehensive evaluation of the driver's attention, and reduces the hardware equipment requirements for the actual vehicle.

[0168] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0169] In this embodiment, a driver's concentration assessment device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0170] Fig. 9 is a structural block diagram of the driver's concentration evaluation device of this embodiment, such as Fig. 9 As shown, the device includes: a first generating module 10, a second generating module 20, a third generating module 30 and an evaluating module 40;

[0171] The first generating module 10 is used to detect each frame of the driver's head and shoulder image, obtain the driver's head posture angle data, and generate a head posture angle feature map corresponding to each head posture angle data;

[0172] The second generating module 20 is used to obtain the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generate a vehicle posture angle feature map corresponding to each first posture angle data;

[0173] The third generating module 30 is used to extract the first binocular image of the driver from each frame of the head and shoulder image, and generate a gaze point feature map corresponding to each first binocular image;

[0174] The evaluation module 40 is used to perform multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluate the driver's concentration based on the fusion result to obtain the driver's concentration level.

[0175] Through the device provided by this embodiment, each frame of the driver's head and shoulder image is detected to obtain the driver's head posture angle data, and a head posture angle feature map corresponding to each head posture angle data is generated; the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image is obtained, and a vehicle posture angle feature map corresponding to each first posture angle data is generated; the driver's first binocular image is extracted from each frame of the head and shoulder image, and a gaze point feature map corresponding to each first binocular image is generated; the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are multi-modally fused, and the driver's concentration is evaluated based on the fusion result to obtain the driver's concentration level, which solves the problem of being unable to accurately and comprehensively evaluate the driver's attention, realizes accurate and comprehensive evaluation of the driver's attention, and reduces the hardware equipment requirements for the actual vehicle.

[0176] In some of the embodiments, the first generating module 10 is further used to obtain multiple frames of head and shoulder images of the driver; by constraining the local neural domain model, the facial feature points in each frame of the head and shoulder image are detected to obtain the driver's head posture angle data.

[0177] In some of the embodiments, the first generating module 10 is also used to obtain the head pitch angle, head roll angle and head yaw angle in each head posture angle data; generate a first cumulative histogram corresponding to each head pitch angle, a second cumulative histogram corresponding to each head roll angle and a third cumulative histogram corresponding to each head yaw angle; integrate the first cumulative histogram, the second cumulative histogram and the third cumulative histogram to obtain a head posture angle feature map.

[0178] In some of the embodiments, the second generation module 20 is further used to obtain the vehicle pitch angle, vehicle roll angle and vehicle yaw angle in each first attitude angle data; generate a fourth cumulative histogram corresponding to each vehicle pitch angle, a fifth cumulative histogram corresponding to each vehicle roll angle and a sixth cumulative histogram corresponding to each vehicle yaw angle; and integrate the fourth cumulative histogram, the fifth cumulative histogram and the sixth cumulative histogram to obtain a vehicle attitude angle feature map.

[0179] In some of the embodiments, the third generation module 30 is further used to standardize each first binocular image to obtain a corresponding second binocular image; process each second binocular image through a convolutional neural network to obtain a corresponding first gaze vector; convert the first gaze vector from a world coordinate system to a camera coordinate system to obtain a second gaze vector; determine the gaze point of the driver on the virtual field of view corresponding to each second gaze vector, and generate a gaze point feature map corresponding to each gaze point.

[0180] In some of the embodiments, the third generation module 30 is further used to standardize the first distance from the center point of the binocular eyes to the center point of the camera in each first binocular image, and determine the corresponding scaling matrix according to the standardized first distance and the coordinates of the center point of the binocular eyes; determine the camera transformation matrix based on the conversion process of the first binocular image from the camera coordinate system to the world coordinate system, and the scaling matrix; determine the target transformation formula according to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera; and transform the first binocular image based on the target transformation formula to obtain a second binocular image.

[0181] In some of the embodiments, the evaluation module 40 is also used to stack the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map based on the time dimension to obtain a multimodal fusion feature map; and input the multimodal fusion feature map into the concentration classification model for processing to obtain the driver's concentration level.

[0182] In some of the embodiments, the evaluation module 40 is further used to transform the multimodal fusion feature map into one dimension; perform feature extraction on the one-dimensionalized multimodal fusion feature map to obtain a concentration feature quantity; and classify the concentration feature quantity to obtain the driver's concentration level.

[0183] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0184] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0185] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0186] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0187] S1, detecting each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generating a head posture angle feature map corresponding to each head posture angle data;

[0188] S2, obtaining first attitude angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generating a vehicle attitude angle feature map corresponding to each first attitude angle data;

[0189] S3, extracting the first binocular image of the driver from each frame of the head and shoulder image, and generating a gaze point feature map corresponding to each first binocular image;

[0190] S4, performing multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluating the driver's concentration based on the fusion result to obtain the driver's concentration level.

[0191] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0192] In addition, in combination with the driver concentration evaluation method provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the driver concentration evaluation methods in the above embodiments is implemented.

[0193] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0194] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0195] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0196] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for evaluating a driver's concentration, characterized in that: The method comprises: Detecting each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generating a head posture angle feature map corresponding to each head posture angle data; Acquire first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generate a vehicle posture angle feature map corresponding to each of the first posture angle data; Extracting the first binocular image of the driver from each frame of the head-shoulder image, and generating a gaze point feature map corresponding to each of the first binocular images; Wherein, the generating of gaze point feature maps corresponding to each of the first binocular images includes: standardizing the first distance from the center point of both eyes to the center point of the camera in each of the first binocular images, and determining the corresponding scaling matrix according to the first distance after standardization and the coordinates of the center point of both eyes; determining the camera transformation matrix based on the conversion process of the first binocular image from the camera coordinate system to the world coordinate system, and the scaling matrix; determining the target transformation formula according to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera; transforming the first binocular image based on the target transformation formula to obtain a second binocular image; processing each of the second binocular images through a convolutional neural network to obtain a corresponding first gaze vector; transforming the first gaze vector from the world coordinate system to the camera coordinate system to obtain a second gaze vector; determining the gaze point of the driver corresponding to each of the second gaze vectors on the virtual field of view, and generating the gaze point feature map corresponding to each of the gaze points; The head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are subjected to multimodal feature fusion, and the concentration of the driver is evaluated based on the fusion result to obtain the concentration level of the driver.

2. The method for evaluating driver concentration according to claim 1, characterized in that: The detecting of each frame of the driver's head and shoulder image to obtain the driver's head posture angle data includes: Acquire multiple frames of the head and shoulder images of the driver; By constraining the local neural domain model, the facial feature points in each frame of the head and shoulder image are detected to obtain the driver's head posture angle data.

3. The method for evaluating driver concentration according to claim 1, characterized in that: The step of generating a head posture angle feature map corresponding to each head posture angle data comprises: Acquire the head pitch angle, head roll angle and head yaw angle in each of the head posture angle data; Generate a first cumulative histogram corresponding to each of the head pitch angles, a second cumulative histogram corresponding to each of the head roll angles, and a third cumulative histogram corresponding to each of the head yaw angles; The first cumulative histogram, the second cumulative histogram and the third cumulative histogram are integrated to obtain the head posture angle feature map.

4. The method for evaluating driver concentration according to claim 1, characterized in that: The generating of the vehicle attitude angle characteristic graph corresponding to each of the first attitude angle data comprises: Obtaining a vehicle pitch angle, a vehicle roll angle, and a vehicle yaw angle in each of the first attitude angle data; Generate a fourth cumulative histogram corresponding to each of the vehicle pitch angles, a fifth cumulative histogram corresponding to each of the vehicle roll angles, and a sixth cumulative histogram corresponding to each of the vehicle yaw angles; The fourth cumulative histogram, the fifth cumulative histogram and the sixth cumulative histogram are integrated to obtain the vehicle attitude angle feature map.

5. The method for evaluating driver concentration according to claim 1, characterized in that: The multimodal feature fusion of the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map is performed, and the concentration of the driver is evaluated based on the fusion result to obtain the concentration level of the driver, including: Based on the time dimension, the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map are stacked to obtain a multimodal fusion feature map; The multimodal fusion feature map is input into a concentration classification model for processing to obtain the concentration level of the driver.

6. The method for evaluating driver concentration according to claim 5, characterized in that: The step of inputting the multimodal fusion feature map into a concentration classification model for processing to obtain the concentration level of the driver includes: One-dimensionalizing the multimodal fusion feature map; Performing feature extraction on the one-dimensionalized multimodal fusion feature map to obtain a concentration feature value; The concentration feature quantity is classified to obtain the concentration level of the driver.

7. A driver concentration assessment device, characterized in that: The device comprises: a first generating module, a second generating module, a third generating module and an evaluating module; The first generating module is used to detect each frame of the driver's head and shoulder image to obtain the driver's head posture angle data, and generate a head posture angle feature map corresponding to each head posture angle data; The second generating module is used to obtain the first posture angle data of the current vehicle corresponding to each frame of the head and shoulder image, and generate a vehicle posture angle feature map corresponding to each of the first posture angle data; The third generating module is used to extract the first binocular image of the driver from each frame of the head and shoulder image, and generate a gaze point feature map corresponding to each of the first binocular images; The third generation module is further used to standardize the first distance from the center point of the binocular eyes to the center point of the camera in each of the first binocular images, and determine the corresponding scaling matrix according to the first standardized distance and the coordinates of the center point of the binocular eyes; determine the camera transformation matrix based on the conversion process of the first binocular image from the camera coordinate system to the world coordinate system, and the scaling matrix; determine the target transformation formula according to the camera transformation matrix and the real camera parameters and virtual camera parameters of the camera; transform the first binocular image based on the target transformation formula to obtain a second binocular image; process each of the second binocular images through a convolutional neural network to obtain a corresponding first gaze vector; transform the first gaze vector from the world coordinate system to the camera coordinate system to obtain a second gaze vector; determine the gaze point of the driver corresponding to each of the second gaze vectors on the virtual field of view, and generate the gaze point feature map corresponding to each of the gaze points; The evaluation module is used to perform multimodal feature fusion on the head posture angle feature map, the vehicle posture angle feature map and the gaze point feature map, and evaluate the driver's concentration based on the fusion result to obtain the driver's concentration level.

8. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the driver's concentration assessment method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for evaluating the concentration of a driver according to any one of claims 1 to 6 are implemented.

Images