Classification method and device for driving scene, storage medium and program product

Abstract

The invention relates to a classification method and device for a driving scene, a storage medium and a program product, and relates to the technical field of machine learning. The method is executed by a vehicle and comprises the following steps: acquiring an eye video of a driver and environment data outside the vehicle in the driving process of the vehicle; determining eye movement information of the driver based on the eye video; the eye movement information comprises direction information of a fixation point of the driver and blinking information; determining obstacle information around the vehicle based on the vehicle exterior environment data; the obstacle information comprises the type of an obstacle on a road in front of the vehicle and the distance between the obstacle and the vehicle; the obstacle type is a movable obstacle or an immovable obstacle; based on the eye movement information of the driver and the obstacle information, obtaining eye movement characteristics of the driver; the eye movement features are input into a classification model, a classification result output by the classification model is obtained, and the classification result is used for indicating the classification type of the driver.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of machine learning, and in particular, to a classification method, device, storage medium and program product for driving scenarios. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of machine learning models is becoming more and more extensive. For example, machine learning models can be applied to classify drivers in driving scenarios.

[0003] In the related art, based on voice or video, the function of classifying drivers in driving scenarios can be realized through a machine learning model. The process of classifying drivers in driving scenarios is as follows: Obtain the voice or facial video of the driver in the vehicle, input the voice or facial video in the vehicle into the machine learning model, and the machine learning model recognizes and analyzes the voice or facial video, and outputs the classification type of the driver corresponding to the voice or facial video. For example, whether the driver is fatigued or not.

[0004] However, in the solution shown in the above related art, since the classification method for driving scenarios is relatively single, the classification accuracy in driving scenarios is low. Summary of the Invention

[0005] The embodiments of the present application provide a classification method, device, storage medium and program product for driving scenarios, which can improve the classification accuracy for driving scenarios. The technical solution is as follows:

[0006] On the one hand, a classification method for driving scenarios is provided. The method is executed by a vehicle, and the method includes:

[0007] During the driving process of the vehicle, obtain the eye video of the driver and the vehicle external environment data;

[0008] Based on the eye video, determine the eye movement information of the driver; the eye movement information includes the direction information of the fixation point of the driver and the blink information;

[0009] Based on the vehicle external environment data, determine the obstacle information around the vehicle; the obstacle information includes the type of obstacles on the road in front of the vehicle and the distance between the obstacles and the vehicle; the obstacle type is a movable obstacle or an immovable obstacle;

[0010] Based on the eye movement information of the driver and the obstacle information, obtain the eye movement characteristics of the driver;

[0011] Input the eye movement features into a classification model to obtain the classification result output by the classification model; the classification result is used to indicate the classification type of the driver.

[0012] On the other hand, a classification device for a driving scenario is provided. The device includes:

[0013] A data acquisition module, configured to acquire the eye video of the driver and the vehicle external environment data during the driving of the vehicle;

[0014] An eye movement information determination module, configured to determine the eye movement information of the driver based on the eye video; the eye movement information includes the direction information of the fixation point of the driver and the blink information;

[0015] An obstacle information determination module, configured to determine the obstacle information around the vehicle based on the vehicle external environment data; the obstacle information includes the type of obstacles on the road in front of the vehicle and the distance between the obstacles and the vehicle; the type of obstacle is a movable obstacle or an immovable obstacle;

[0016] An eye movement feature acquisition module, configured to acquire the eye movement features of the driver based on the eye movement information of the driver and the obstacle information;

[0017] A classification result acquisition module, configured to input the eye movement features into a classification model to obtain the classification result output by the classification model; the classification result is used to indicate the classification type of the driver.

[0018] In some embodiments, the eye movement information determination module is configured to input the eye video into an action recognition model to obtain the pupil information and the blink information of the driver output by the action recognition model; the pupil information includes the position of the pupil; the blink information includes the blink frequency and the blink speed;

[0019] Determine the direction information of the fixation point of the driver based on the pupil information of the driver.

[0020] In some embodiments, the eye movement information determination module is configured to obtain the first position and the second position of the driver's pupil within a first period of time; the first position is the position of the driver's pupil at a first moment; the second position is the position of the driver's pupil at a second moment; the first moment and the second moment are adjacent moments;

[0021] Obtain a first time difference, a second time difference, and an angular difference; the first time difference is the time difference between the duration when the infrared camera of the vehicle emits infrared rays to a first position when the driver's pupil is at the first position and the duration when the infrared rays are reflected back from the first position to the infrared camera; the second time difference is the time difference between the duration when the infrared camera of the vehicle emits infrared rays to a second position when the driver's pupil is at the second position and the duration when the infrared rays are reflected back from the second position to the infrared camera; the angular difference is the change angle between the infrared rays reflected back from the first position and the infrared rays reflected back from the second position received by the infrared camera.

[0022] Based on the first time difference, the second time difference, and the angular difference, determine the displacement distance of the driver's pupil.

[0023] Based on the displacement distance of the driver's pupil and the position of the driver's pupil, determine the displacement angle of the driver's pupil.

[0024] Through the displacement distance and the displacement angle, query and obtain the direction information of the driver's fixation point from the relationship among the displacement distance, the displacement angle, and the direction of the driver's fixation point.

[0025] In some embodiments, the eye movement information determination module is configured to determine the displacement distance of the driver's pupil determined based on the first time difference, the second time difference, and the angular difference through the following formula (1) based on the first time difference, the second time difference, and the angular difference:

[0026] Formula (1):

[0027] Wherein, C represents the speed of light, T1 represents the first time difference, T2 represents the second time difference, and θ1 represents the angular difference.

[0028] In some embodiments, the obstacle information determination module is configured to input the vehicle external environment data into an obstacle recognition model and obtain the obstacle information around the vehicle output by the obstacle recognition model.

[0029] In some embodiments, the eye movement features include blink frequency, blink speed, close obstacle attention rate, far obstacle attention rate, proportion of fixation time on close obstacles, proportion of fixation time on far obstacles, small obstacle attention rate, proportion of fixation time on small obstacles, large obstacle attention rate, proportion of fixation time on large obstacles, proportion of fixation time on movable obstacles, movable obstacle attention rate, immovable obstacle attention rate, proportion of fixation time on immovable obstacles, average fixation distance, maximum fixation distance value, minimum fixation distance value, median fixation distance, variance of fixation distance, average eye movement rate, maximum eye movement rate, median eye movement rate, variance of eye movement rate, saccade count, saccade time, fixation spatial density, and total fixation duration.

[0030] In another aspect, a computer device is provided. The computer device includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the classification method for driving scenarios as described above.

[0031] In another aspect, a computer-readable storage medium is provided. At least one instruction, at least one program, a code set, or an instruction set is stored in the storage medium. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the classification method for driving scenarios as described above.

[0032] In still another aspect, a computer program product is provided. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the classification method for driving scenarios provided in the above various optional implementation manners.

[0033] The technical solution provided by this application may include the following beneficial effects:

[0034] During the driving process of the vehicle, the eye video of the driver and the vehicle external environment data can be obtained, the eye movement information and the obstacle information around the vehicle can be determined. The eye movement information can directly reflect the attention distribution of the driver, and the obstacle information can determine different types of obstacles faced by the driver. By combining the eye movement information with the obstacle information to obtain the eye movement features of the driver, the reaction state of the driver in different driving scenarios can be captured. Inputting the eye movement features into the classification model, the classification model can more accurately identify the corresponding type of the driver based on the subtle reactions of the driver facing different obstacles, effectively improving the accuracy of the classification for driving scenarios.

[0035] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0037] Figure 1 is a system configuration diagram of a classification method for driving scenarios according to an embodiment of this application;

[0038] Figure 2 is a flowchart of a classification method for driving scenarios provided by an embodiment of this application;

[0039] Figure 3 is a flowchart of a classification method for driving scenarios provided by an embodiment of this application;

[0040] Figure 4 is a flowchart of a classification method for driving scenarios provided by an embodiment of this application;

[0041] Figure 5 is a flowchart of a classification method for driving scenarios provided by an embodiment of this application;

[0042] Figure 6 is a development flowchart of a classification model for driving scenarios provided by an embodiment of this application;

[0043] Figure 7 is a schematic structural diagram of a classification device for driving scenarios provided by an embodiment of this application;

[0044] Figure 8 is a block diagram of a classification device for driving scenarios provided by an exemplary embodiment of this application;

[0045] Figure 9 is a schematic structural diagram of a computer device provided by an exemplary embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0047] On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] Figure 1 This is a system configuration diagram of a classification method for driving scenarios according to an embodiment of the present application. As Figure 1 shown, vehicle 100 includes a classification model 100a. During the driving process of vehicle 100, vehicle 100 acquires the eye video of the driver and the external environment data of the vehicle; based on the eye video, the eye movement information of the driver is determined; the eye movement information includes the direction information of the driver's fixation point and the blinking information; based on the external environment data, the obstacle information around vehicle 100 is determined; the obstacle information includes the type of obstacles on the road in front of vehicle 100 and the distance between the obstacles and vehicle 100; the obstacle type is a movable obstacle or an immovable obstacle; vehicle 100 acquires the eye movement characteristics of the driver based on the eye movement information and the obstacle information of the driver; the eye movement characteristics are input into the classification model 100a, and the classification result output by the classification model 100a is obtained; the classification result is used to indicate the classification type of the driver.

[0049] Exemplarily, please refer to Figure 2 , Figure 2 This is a flowchart of a classification method for driving scenarios provided by an embodiment of the present application. Among them, the classification method for driving scenarios can be executed by a vehicle. For example, the vehicle can be the above-mentioned Figure 1 vehicle 100 shown. The above-mentioned classification method for driving scenarios may include step 210, step 220, step 230, step 240, and step 250, and the specific implementation is as follows.

[0050] Step 210: During the driving process of the vehicle, acquire the eye video of the driver and the external environment data of the vehicle.

[0051] Among them, the above-mentioned eye video of the driver is a continuous image sequence of the driver's eye area during the driving process of the vehicle.

[0052] In the embodiment of the present application, the above-mentioned eye video of the driver can be acquired through a sensor (for example, a camera) of the vehicle. In some embodiments, the camera can be installed on the dashboard inside the vehicle. During the driving process of the vehicle, the camera continuously captures a continuous image sequence of the driver's eyes to generate the eye video of the driver.

[0053] Among them, the above-mentioned external environment data of the vehicle is the environmental data around the vehicle during the driving process of the vehicle. For example, the environmental data can indicate whether there are stationary or moving obstacles around the vehicle.

[0054] In the embodiment of the present application, the above-mentioned external environment data of the vehicle can be acquired through sensors (for example, lidar, millimeter-wave radar, external camera) of the vehicle. For example, the above-mentioned external environment data can include at least one of radar point cloud data and video image data.

[0055] In some embodiments, when the vehicle speed meets the first vehicle speed, the vehicle acquires the eye video of the driver and the vehicle external environment data; when the vehicle speed meets the second vehicle speed, the acquisition of the eye video of the driver and the vehicle external environment data is stopped; the first vehicle speed is less than the second vehicle speed.

[0056] Wherein, the above-mentioned first vehicle speed is a vehicle speed when the vehicle is driving at a low speed, and the above-mentioned second vehicle speed is a vehicle speed when the vehicle is driving at a relatively high speed. For example, the above-mentioned first vehicle speed can be a vehicle speed greater than 0 km / h and less than 30 km / h; the above-mentioned second vehicle speed can be a vehicle speed higher than 30 km / h.

[0057] In the embodiments of the present application, the vehicle can monitor the vehicle speed in real time through a speed sensor, and judge whether the vehicle speed meets the first vehicle speed / second vehicle speed through the vehicle control system. When the vehicle speed meets the first vehicle speed, the vehicle control system controls the on-vehicle camera to acquire the eye video of the driver and the external environment data; when the vehicle speed rises to meet the second vehicle speed, an instruction is sent to the on-vehicle camera through the vehicle control system, so that the on-vehicle camera stops acquiring the eye video of the driver and the vehicle external environment data.

[0058] In the embodiments of the present application, when the vehicle is driving at a low speed (i.e., driving at the first vehicle speed), the vehicle acquires the eye video of the driver and the vehicle external environment data to ensure safety and optimize the assisted driving function. When the vehicle speed reaches or exceeds a relatively high driving state (i.e., driving at the second vehicle speed), the vehicle stops acquiring the eye video of the driver and the vehicle external environment data to reduce the unnecessary computing burden of the vehicle. In the low-speed state, the road conditions are usually more complex. Correspondingly, the user's observation of the vehicle exterior will be more frequent and the observation directions will be more (such as frequent observation in all directions: front, back, left, and right). At this time, the collected data can better reflect the driver's observation behavior. On the contrary, in the high-speed state, the road conditions are usually relatively simple (such as elevated roads or highways). Correspondingly, the user's observation of the vehicle exterior will not be too frequent and the observation directions will not be too many (such as at this time the driver will mostly only observe the front and will only observe the side when changing lanes). At this time, the collected data usually cannot well reflect the driver's observation behavior; the above embodiments of the present application collect data at a lower vehicle speed, which can improve the quality of the collected data and ensure the accuracy of subsequent classification.

[0059] Step 220: Determine the eye movement information of the driver based on the eye video; the eye movement information includes the direction information of the driver's fixation point and the blink information.

[0060] Wherein, the above-mentioned direction information of the driver's fixation point refers to the fixation direction of the driver's eyes, and the fixation direction of the driver's eyes can reflect the focus of the driver's attention.

[0061] In an embodiment of the present application, a vehicle may input an eye video into a machine learning model to obtain direction information of a driver's fixation point output by the machine learning model, where the machine learning model is a machine learning model capable of obtaining direction information of the driver's fixation point based on the eye video.

[0062] Among them, the above blink information may include the blink frequency and blink speed of the driver.

[0063] In an embodiment of the present application, a vehicle may determine the driver's eye movement information based on an eye video through a convolutional neural network, that is, perform feature extraction and classification on the driver's eye region through the convolutional neural network, and calculate the blink frequency and blink speed by statistically analyzing the time intervals and durations of blink events.

[0064] Step 230: Determine obstacle information around the vehicle based on the external environment data of the vehicle; the obstacle information includes the type of obstacles on the road in front of the vehicle and the distance between the obstacles and the vehicle; the type of obstacles is a movable obstacle or an immovable obstacle.

[0065] In an embodiment of the present application, after obtaining the external environment data of the vehicle, the vehicle may determine the obstacle information around the vehicle through an object detection algorithm, where the object detection algorithm is an algorithm capable of identifying obstacles based on the external environment data of the vehicle and obtaining the distance information between the obstacles and the vehicle in combination with the vehicle radar.

[0066] Step 240: Obtain the driver's eye movement characteristics based on the driver's eye movement information and the obstacle information.

[0067] Among them, the above eye movement characteristics include but are not limited to blink frequency, blink speed, the attention rate of the driver to different types of obstacles, and the fixation time ratio of different types of obstacles.

[0068] In an embodiment of the present application, the vehicle may determine the attention rate of the driver to a specific type of obstacle based on the fixation duration and the number of fixations of the driver on the specific type of obstacle.

[0069] In an embodiment of the present application, the vehicle may determine the fixation duration and the number of fixations of the driver on different types of obstacles through the driver's eye movement information; different types of obstacles may be determined through the obstacle information, and the attention rate and fixation time ratio of different types of obstacles may be determined through the driver's fixation behavior on specific obstacles.

[0070] For example, it is determined through the eye movement information that the fixation duration of the driver on type A obstacles is T1, and the fixation duration on all obstacles is T2. The fixation time ratio of the driver on type A obstacles can be calculated through T1 and T2 as

[0071] Step 250: Input the eye movement features into the classification model to obtain the classification result output by the classification model; the classification result is used to indicate the classification type of the driver.

[0072] Among them, the above classification model is a machine learning model that can predict the classification type of the driver according to the eye movement features.

[0073] In the embodiment of the present application, the vehicle can directly obtain the predicted type corresponding to the driver through the classification model; optionally, the vehicle inputs the eye movement features into the classification model, and the classification model outputs a probability distribution, which represents the confidence that the driver belongs to different types, and obtains the category with the highest confidence as the classification type of the driver.

[0074] In the embodiment of the present application, the above classification types of the driver include but are not limited to driving style types, attention concentration levels, fatigue levels, and personality types, etc.

[0075] Optionally, the vehicle inputs the eye movement features into the driving style classification model to obtain the driving style classification result output by the driving style classification model; the driving style classification result is used to indicate the driving style type of the driver.

[0076] Among them, the above driving style classification model is a machine learning model that can predict the driving style type of the driver according to the eye movement features.

[0077] Among them, the above driving style type can be one of a steady type, an aggressive type, and a cautious type.

[0078] Optionally, the vehicle inputs the eye movement features into the attention test model to obtain the attention concentration level output by the attention test model; the attention concentration level is used to indicate the attention concentration level of the driver during driving.

[0079] Among them, the above attention test model is a machine learning model that can predict the attention concentration level of the driver according to the eye movement features.

[0080] Among them, the above attention concentration level can be one of highly concentrated, moderately concentrated, and distracted.

[0081] Optionally, the vehicle inputs the eye movement features into the fatigue level classification model to obtain the fatigue level classification result output by the fatigue level classification model; the fatigue level classification result is used to indicate the fatigue level of the driver.

[0082] Among them, the above fatigue level classification model is a machine learning model that can predict the fatigue level of the driver according to the eye movement features.

[0083] Among them, the above fatigue level can be one of mild fatigue, moderate fatigue, and severe fatigue.

[0084] Optionally, the vehicle inputs the eye movement features into the personality classification model to obtain the personality classification result output by the personality classification model; the personality classification result is used to indicate the personality type of the driver.

[0085] Among them, the above-mentioned personality classification model is a machine learning model that can predict the personality type of the driver according to the eye movement features.

[0086] Among them, the above-mentioned personality type of the driver can be one of neuroticism, extraversion, openness, agreeableness, and conscientiousness.

[0087] In the embodiment of the present application, during the driving process of the vehicle, the eye video of the driver and the vehicle external environment data can be obtained to determine the eye movement information and the obstacle information around the vehicle. The eye movement information can directly reflect the attention distribution of the driver, and the obstacle information can determine different types of obstacles faced by the driver. By combining the eye movement information with the obstacle information to obtain the eye movement features of the driver, the reaction state of the driver in different driving scenarios can be captured. Inputting the eye movement features into the classification model, the classification model can more accurately identify the corresponding type of the driver based on the subtle reactions of the driver facing different obstacles, effectively improving the accuracy of classification for driving scenarios.

[0088] Based on the solutions shown in any one or more of the above embodiments, in some embodiments, the eye movement features include blink frequency, blink speed, close obstacle attention rate, far obstacle attention rate, close obstacle fixation time ratio, far obstacle fixation time ratio, small obstacle attention rate, small obstacle fixation time ratio, large obstacle attention rate, large obstacle fixation time ratio, movable obstacle fixation time ratio, movable obstacle attention rate, immovable obstacle attention rate, immovable obstacle fixation time ratio, average fixation distance, maximum fixation distance value, minimum fixation distance value, fixation distance median, fixation distance variance, average eye movement rate, maximum eye movement rate, eye movement rate median, eye movement rate variance, saccade count, saccade time, fixation spatial density, and total fixation duration.

[0089] Among them, the above-mentioned blink frequency is the number of times the driver blinks per unit time (for example, per minute).

[0090] In the embodiment of the present application, the vehicle can directly obtain the number of blinks of the driver within a specified duration through the eye movement information of the driver to obtain the blink frequency.

[0091] Among them, the above-mentioned blink speed refers to the time from the start of eye closure to the complete opening of the eyes each time the driver blinks.

[0092] In an embodiment of the present application, the vehicle can obtain the timestamps of the driver's eyes at the moment of starting to close and fully opening when blinking through eye movement information, and determine the blinking speed based on these two timestamps.

[0093] Among them, the above-mentioned close-range obstacle attention rate refers to the proportion of the number of times the driver gazes at obstacles within the first distance to the total number of times of gazing at all obstacles; the long-range obstacle attention rate refers to the proportion of the number of times the driver gazes at obstacles within the second distance to the total number of times of gazing at all obstacles; the first distance is less than the second distance.

[0094] In an embodiment of the present application, the vehicle can determine the distance between the vehicle and the close-range obstacle through obstacle information, combine the eye movement information, count the number of times the driver gazes at the close-range obstacle, and calculate the proportion of the number of times the driver gazes at the close-range obstacle to the total number of times the driver gazes at all obstacles as the close-range obstacle attention rate.

[0095] In an embodiment of the present application, the vehicle can determine the distance between the vehicle and the long-range obstacle through obstacle information, combine the eye movement information, count the number of times the driver gazes at the long-range obstacle, and calculate the proportion of the number of times the driver gazes at the long-range obstacle to the total number of times the driver gazes at all obstacles as the close-range obstacle attention rate.

[0096] Among them, the above-mentioned close-range obstacle gazing time proportion is the proportion of the duration that the driver's line of sight stays on the obstacle within the first distance to the total duration of gazing at all obstacles; the long-range obstacle gazing time proportion is the proportion of the duration that the driver's line of sight stays on the obstacle within the second distance to the total duration of gazing at all obstacles.

[0097] In an embodiment of the present application, the vehicle can determine the distance between the vehicle and the close-range obstacle through obstacle information, combine the eye movement information, count the total duration that the driver gazes at the close-range obstacle, and calculate the proportion of the total duration that the driver gazes at the close-range obstacle to the total duration that the driver gazes at all obstacles as the close-range obstacle gazing time proportion.

[0098] In an embodiment of the present application, the vehicle can determine the distance between the vehicle and the long-range obstacle through obstacle information, combine the eye movement information, count the total duration that the driver gazes at the long-range obstacle, and calculate the proportion of the total duration that the driver gazes at the long-range obstacle to the total duration that the driver gazes at all obstacles as the long-range obstacle gazing time proportion.

[0099] Among them, the above-mentioned small obstacle attention rate is the proportion of the number of times the driver gazes at obstacles of the first size to the total number of times of gazing at all obstacles; the small obstacle gaze time proportion is the proportion of the duration that the driver's line of sight stays on obstacles of the first size to the total duration of gazing at all obstacles; the large obstacle attention rate is the proportion of the number of times the driver gazes at obstacles of the second size to the total number of times of gazing at all obstacles; the large obstacle gaze time proportion is the proportion of the duration that the driver's line of sight stays on obstacles of the second size to the total duration of gazing at all obstacles; the first size is smaller than the second size.

[0100] In the embodiment of the present application, the vehicle can determine small obstacles with a size meeting the first size through obstacle information, count the number of times the driver gazes at the small obstacles in the eye movement information, and calculate the proportion of the number of times the driver gazes at the small obstacles to the total number of times the driver gazes at all obstacles as the small obstacle attention rate.

[0101] In the embodiment of the present application, the vehicle can determine small obstacles with a size meeting the first size through obstacle information, count the gazing duration of the driver on the small obstacles in the eye movement information, calculate the total duration that the driver gazes at the small obstacles, and calculate the proportion of the total duration that the driver gazes at the small obstacles to the total duration that the driver gazes at all obstacles as the small obstacle gaze time proportion.

[0102] In the embodiment of the present application, the vehicle can determine large obstacles with a size meeting the second size through obstacle information, count the number of times the driver gazes at the large obstacles in the eye movement information, and calculate the proportion of the number of times the driver gazes at the large obstacles to the total number of times the driver gazes at all obstacles as the large obstacle attention rate.

[0103] In the embodiment of the present application, the vehicle can determine large obstacles with a size meeting the second size through obstacle information, count the gazing duration of the driver on the large obstacles in the eye movement information, calculate the total duration that the driver gazes at the large obstacles, and calculate the proportion of the total duration that the driver gazes at the large obstacles to the total duration that the driver gazes at all obstacles as the large obstacle gaze time proportion.

[0104] Among them, the above-mentioned movable obstacle gaze time proportion is the proportion of the duration that the driver's line of sight stays on the movable obstacles to the total duration of gazing at all obstacles; the movable obstacle attention rate is the proportion of the number of times the driver gazes at the movable obstacles to the total number of times of gazing at all obstacles; the immovable obstacle attention rate is the proportion of the number of times the driver gazes at the immovable obstacles to the total number of times of gazing at all obstacles; the immovable obstacle gaze time proportion is the proportion of the duration that the driver's line of sight stays on the immovable obstacles to the total duration of gazing at all obstacles.

[0105] In the embodiments of the present application, the vehicle can determine movable obstacles through obstacle information, count the number of times the driver gazes at the movable obstacles in the eye movement information, calculate the proportion of the number of times the driver gazes at the movable obstacles to the total number of times the driver gazes at all obstacles, and use it as the movable obstacle attention rate.

[0106] In the embodiments of the present application, the vehicle can determine movable obstacles through obstacle information, count the duration of the driver's gaze at the movable obstacles in the eye movement information, calculate the total duration of the driver's gaze at the movable obstacles, calculate the proportion of the total duration of the driver's gaze at the movable obstacles to the total duration of the driver's gaze at all obstacles, and use it as the movable obstacle gaze time ratio.

[0107] In the embodiments of the present application, the vehicle can determine immovable obstacles through obstacle information, count the number of times the driver gazes at the immovable obstacles in the eye movement information, calculate the proportion of the number of times the driver gazes at the immovable obstacles to the total number of times the driver gazes at all obstacles, and use it as the immovable obstacle attention rate.

[0108] In the embodiments of the present application, the vehicle can determine immovable obstacles through obstacle information, count the duration of the driver's gaze at the immovable obstacles in the eye movement information, calculate the total duration of the driver's gaze at the immovable obstacles, calculate the proportion of the total duration of the driver's gaze at the immovable obstacles to the total duration of the driver's gaze at all obstacles, and use it as the immovable obstacle gaze time ratio.

[0109] Among them, the above average gaze distance is the average value of the distance between the driver's gaze point and the vehicle each time; the maximum gaze distance value refers to the maximum distance value between the driver's gaze point and the vehicle; the minimum gaze distance value is the minimum distance value between the driver's gaze point and the vehicle.

[0110] In the embodiments of the present application, the vehicle obtains the distance between the obstacle and the vehicle through obstacle information, combines the gaze point of the driver indicated by the eye movement information, calculates the distance between the driver's gaze point and the vehicle each time and takes the average value, as the average gaze distance.

[0111] In the embodiments of the present application, the vehicle obtains the distance between the obstacle and the vehicle through obstacle information, combines the gaze point of the driver indicated by the eye movement information, calculates the distance between the driver's gaze point and the vehicle each time and obtains the maximum distance among them, as the maximum gaze distance value.

[0112] In the embodiments of the present application, the vehicle obtains the distance between the obstacle and the vehicle through obstacle information, combines the gaze point of the driver indicated by the eye movement information, calculates the distance between the driver's gaze point and the vehicle each time and obtains the minimum distance among them, as the minimum gaze distance value.

[0113] Among them, for all the fixation events of the driver with the median fixation distance, the distance value at the middle position after sorting the distances from smallest to largest.

[0114] In the embodiments of the present application, the vehicle obtains the distance between the obstacle and the vehicle through obstacle information, combines the fixation points of the driver indicated by the eye movement information, calculates the distance between each fixation point of the driver and the vehicle, sorts them from smallest to largest, and obtains the median value as the median fixation distance.

[0115] Among them, the above-mentioned fixation space density is the distribution pixel density of the driver's fixation points in the road space within a specified range.

[0116] Among them, the above-mentioned total fixation duration is the fixation duration of the driver on all obstacles.

[0117] In the embodiments of the present application, a variety of different types of eye movement features are extended. The eye movement features include multi-dimensional data such as blink frequency, blink speed, attention rates to obstacles of different distances and types, and fixation time ratios. The multi-level eye movement features can comprehensively reflect the attention distribution of the driver in different driving environments. Inputting more refined eye movement features into the classification model can generate more accurate classification results, effectively improving the accuracy of classification for driving scenarios.

[0118] Based on Figure 2 , please refer to Figure 3 , Figure 3 FIG.

[0119] Step 220a: Input the eye video into the action recognition model to obtain the pupil information and blink information of the driver output by the action recognition model; the pupil information includes the position of the pupil; the blink information includes the blink frequency and blink speed.

[0120] Among them, the above-mentioned action recognition model is a deep learning model capable of obtaining pupil information and blink information according to the eye video.

[0121] Among them, the above-mentioned pupil information includes the position of the pupil, and the position of the pupil includes the positions of the driver's pupils at different moments, for example, the positions before and after the driver's pupil moves.

[0122] Step 220b: Based on the pupil information of the driver, determine the direction information of the driver's fixation point.

[0123] Among them, the above-mentioned direction information of the fixation point is the line-of-sight direction of the driver's eyes.

[0124] In the embodiment of the present application, based on the pupil information of the driver, the vehicle can determine the direction information of the driver's fixation point by querying a mapping table, which contains the mapping relationship between the pupil information of the driver and the direction information of the driver's fixation point.

[0125] In the embodiment of the present application, the vehicle inputs the eye video into an action recognition model to obtain the pupil position and blink information of the driver, including blink frequency and blink speed, and further determines the direction information of the fixation point based on the pupil information. Since the pupil information can capture the subtle eye movement characteristics of the driver, based on the pupil information of the driver, the accuracy of obtaining the direction information of the driver's fixation point can be effectively guaranteed.

[0126] Based on the solutions shown in any one or more of the above embodiments, in some embodiments, step 220b may be implemented as follows: obtaining the first position and the second position of the driver's pupil within a first period of time; the first position is the position of the driver's pupil at a first moment; the second position is the position of the driver's pupil at a second moment; the first moment and the second moment are adjacent moments; obtaining a first time difference, a second time difference, and an angular difference; the first time difference is the time difference between the duration of the infrared camera of the vehicle emitting infrared rays to the first position and the duration of the infrared rays reflecting back from the first position to the infrared camera when the driver's pupil is at the first position; the second time difference is the time difference between the duration of the infrared camera of the vehicle emitting infrared rays to the second position and the duration of the infrared rays reflecting back from the second position to the infrared camera when the driver's pupil is at the second position; the angular difference is the change angle between the infrared rays reflected back from the first position and the infrared rays reflected back from the second position received by the infrared camera; determining the displacement distance of the driver's pupil based on the first time difference, the second time difference, and the angular difference; determining the displacement angle of the driver's pupil based on the displacement distance of the driver's pupil and the position of the driver's pupil; and querying and obtaining the direction information of the driver's fixation point from the relationship among the displacement distance, the displacement angle, and the direction of the driver's fixation point through the displacement distance and the displacement angle.

[0127] In the embodiment of the present application, the vehicle can directly obtain the first position and the second position of the driver's pupil from the pupil information of the driver.

[0128] In an embodiment of the present application, the vehicle can record the infrared rays emitted by the infrared camera of the vehicle, record the time when the infrared rays are emitted to the first position / second position, and the time when the infrared rays are reflected back from the first position / second position to the infrared camera, and calculate the round-trip time difference of the infrared rays as the first time difference / second time difference when the driver's pupil is at the first position / second position respectively, that is, calculate the difference between the time when the infrared rays are emitted from the infrared camera to the first position and the time when the infrared rays are reflected back from the first position to the infrared camera as the first time difference; calculate the difference between the time when the infrared rays are emitted from the infrared camera to the second position and the time when the infrared rays are reflected back from the second position to the infrared camera as the second time difference.

[0129] In an embodiment of the present application, the vehicle can convert the first position and the second position into coordinate points on the image plane, and based on the direction vectors of the coordinate points on the image plane relative to the center of the infrared camera, calculate the included angle between these two direction vectors through the included angle formula between vectors, and then calculate the change angle between the infrared rays reflected back from the first position and the infrared rays reflected back from the second position as the angle difference through the arccosine function.

[0130] In an embodiment of the present application, the vehicle can calculate the time required for the infrared rays to travel back and forth to the position of the driver's pupil based on the first time difference and the second time difference recorded by the infrared camera, estimate the distance between the driver's pupil and the infrared camera, and then calculate the displacement distance of the driver's pupil through the cosine theorem based on the angle difference.

[0131] In an embodiment of the present application, after the vehicle determines the displacement distance of the driver's pupil, it combines the first position and the second position of the driver's pupil to determine the displacement angle of the driver's pupil. Assuming that the coordinates of the pupil at the first moment and the second moment are P1(x1, y1) and P2(x2, y2) respectively, the displacement vector can be expressed as V=(x2 - x1, y2 - y1), and the displacement angle of the driver's pupil is calculated through the arctangent function.

[0132] In an embodiment of the present application, a mapping table can be stored inside the vehicle. The mapping table contains the mapping relationship between the displacement distance and displacement angle of the driver's pupil and the direction of the driver's gaze point. After the vehicle determines the displacement distance and displacement angle of the driver's pupil, it queries the mapping table to obtain the direction of the driver's gaze point corresponding to the displacement distance and displacement angle in the mapping table.

[0133] In the embodiments of the present application, the vehicle determines the specific movement of the driver's pupil by recording the first position and the second position of the driver's pupil at adjacent moments, and combining the time differences (the first time difference and the second time difference) and the angle difference of the infrared reflection. By querying the relationship between the displacement distance, the displacement angle and the direction of the fixation point, high-precision fixation point direction information is obtained, which can effectively ensure the accuracy of obtaining the direction information of the driver's fixation point.

[0134] Based on the solutions shown in any one or more of the above embodiments, in some embodiments, based on the first time difference, the second time difference, and the angle difference, the displacement distance of the driver's pupil determined based on the first time difference, the second time difference, and the angle difference is determined by the following formula (1):

[0135] Formula (1):

[0136] where C represents the speed of light, T1 represents the first time difference, T2 represents the second time difference, and θ1 represents the angle difference.

[0137] In the embodiments of the present application, the displacement distance of the driver's pupil is calculated by Formula (1). Through high-precision calculation, the minute displacement of the driver's pupil can be accurately captured, effectively ensuring the accuracy of calculating the displacement distance of the driver's pupil.

[0138] Based on Figure 2 , please refer to Figure 4 , Figure 4 FIG. 22 is a flowchart of a classification method for a driving scenario provided by an embodiment of the present application. In some embodiments, the above step 230 may be implemented as step 230a:

[0139] Step 230a: Input the vehicle external environment data into an obstacle recognition model, and obtain the obstacle information around the vehicle output by the obstacle recognition model.

[0140] Wherein, the above obstacle recognition model is a machine learning model capable of determining the obstacle information around the vehicle according to the vehicle external environment data.

[0141] Wherein, the above obstacle information includes the type of the obstacle (for example, a movable obstacle or an immovable obstacle), the distance between the obstacle and the vehicle, and the orientation of the obstacle relative to the vehicle.

[0142] In the embodiments of the present application, the vehicle external environment data includes the environmental data around the vehicle. The vehicle inputs the vehicle external environment data into the obstacle recognition model, and different types of obstacle information are recognized by the obstacle recognition model, effectively ensuring the accuracy of obtaining the obstacle information around the vehicle.

[0143] Based on the above Figures 2 to 4In the steps of the embodiments, the embodiments of the present application disclose a personality recognition method and an automotive structure. Exemplarily, please refer to Figure 5 , Figure 5 which is a flowchart of a personality recognition method provided by an embodiment of the present application, including steps S1, S2, S3, S4, S5, S6, S7, and S8, as follows:

[0144] In the embodiments of the present application, the automotive structure mainly includes the following parts: 1. Physical layer: millimeter-wave radar, external camera, in-vehicle infrared camera, computing and memory unit. 2. Communication layer: responsible for signal analysis and transmission. 3. Data layer: analyzes the signals of the camera, millimeter-wave radar, and infrared camera and conducts analysis. 4. Decision layer: responsible for data computing and model loading to recognize personality traits.

[0145] Step S1: Start when the vehicle speed is higher than 0 and lower than 30 km / h.

[0146] In the embodiments of the present application, when the vehicle speed of the vehicle is higher than 0 km / h and lower than 30 km / h, the vehicle calls the road information of the millimeter-wave radar and the external camera, and starts the in-vehicle infrared camera to record the changes in the driver's eyes; when the vehicle speed is higher than 30 km / h, the vehicle stops calling the external road information.

[0147] Step S2: Sense the road conditions in front of the driver through the millimeter-wave radar and the external camera.

[0148] In the embodiments of the present application, the vehicle can obtain external road information through the millimeter-wave radar and the external camera and use the external road information as the stored information Sig_road.

[0149] Exemplarily, the millimeter-wave radar of the vehicle sends high-frequency millimeter-wave signals to the target through the antenna, calculates the distance between the target and the radar by measuring the round-trip time of the signals, analyzes the speed information of the target's movement using the Doppler effect. After these signals interact with the target, the reflected signals are received by the receiver, and a three-dimensional position image of the target is generated based on the intensity and phase information of the reflected signals.

[0150] In the embodiments of the present application, the vehicle can obtain the position information of each point of the three-dimensional image of the road obstacle in front through millimeter-wave identification, and can obtain the approximate contour, size, and distance of the obstacle. The type of the obstacle can be recognized through the approximate contour of the obstacle.

[0151] In the embodiments of the present application, the distance between each obstacle and the vehicle can be calculated by the following formula two.

[0152] Formula two: D = C × T / 2

[0153] Among them, C is the speed of light, T is the total time from the departure to the reception of the millimeter wave, and the distance between the obstacle and the vehicle is the minimum distance from the vehicle to all points on the obstacle.

[0154] In the embodiment of the present application, the size of the obstacle is represented by the recognized length of the obstacle. The length of the obstacle is defined as the maximum distance between two points of the obstacle. Search for all the emission light angles on a single obstacle, find the maximum angle θ, and record the distances Da and Db of the corresponding contour points. The calculation formula for the length L of the obstacle can be realized by the following formula three:

[0155] Formula three:

[0156] Through an external camera, obtain the image of the road ahead. Here, only an example of a binocular camera is introduced.

[0157] Assume that the centers of the left and right cameras of the camera are B, and there is an imaging point 1 and 2 on each of the left and right cameras at the position point. The distance from the imaging point 1 to the left edge of the left camera is a, and the distance from the imaging point 2 to the left edge of the right lens is b. The focal lengths of both cameras are f. Then the distance Z from the position point to the vehicle can be calculated by the following formula four:

[0158] Formula four:

[0159] Step S3: The infrared camera is started to record the steering information of the driver's pupils.

[0160] In the embodiment of the present application, the in-vehicle infrared camera of the vehicle is started, and the steering information of the driver's pupils can be recognized, and the steering information of the driver's pupils is recorded as the stored information Sig_eyes.

[0161] Step S4: Start the calculation work of the data layer and the decision layer when the vehicle speed is higher than 30 km / h.

[0162] In the embodiment of the present application, when the vehicle speed of the vehicle is higher than 30 km / h, the calculation work of the data layer and the decision layer is started.

[0163] Step S5: Identify the distance and type of obstacles on the current road through the currently mature obstacle recognition neural network model.

[0164] In the embodiment of the present application, through the currently mature obstacle recognition neural network model, based on the road information Sig_road recorded in step S1 and the distance and type of obstacles on the front road (classify obstacles according to whether they are movable, for example: cars and people are classified as movable obstacles, flower beds and trash cans are classified as immovable obstacles).

[0165] Step S6: Record the rotation angle of the driver's pupil through an infrared camera and calculate the direction of the point the driver is looking at.

[0166] Among them, the above infrared camera usually includes an infrared emitter and a receiver. The emitter emits infrared rays to irradiate an object, and after the infrared rays are diffusely reflected, they are received by the receiver to form a video image.

[0167] In the embodiment of the present application, the vehicle can record the displacement of the driver's pupil at two adjacent times by reflecting light.

[0168] For example: two adjacent times t1 and t2. At time t1, the position of the pupil is point A, and at time t2, the position of the pupil is point B. Record the time differences T1 and T2 of the two infrared emissions and receptions, and record the angle difference θ1 when the two lights are received. The displacement distance De of the pupil can be calculated through the above formula (1).

[0169] Formula (1):

[0170] C is the speed of light.

[0171] Through the position of the pupil, the position information of the fixation point can be simulated through experiments.

[0172] Step S7: Overlap the direction information of the fixation point and the current road condition information obtained in S1, and extract the driver's eye movement characteristics.

[0173] In the embodiment of the present application, the vehicle can overlap the direction information of the driver's fixation point and the current road condition information obtained in step S1, and extract the driver's eye movement characteristics.

[0174] Among them, there are 27 eye movement features F1 - F27 in total: (F1) the blinking frequency of the driver (the number of times the eyes are closed recorded by the infrared camera), (F2) the blinking speed (the speed reached when the eyelids close and open), (F3) the proportion of the time the driver gazes at nearby obstacles (within 5m), (F4) the proportion of the number of times the driver gazes at nearby obstacles (within 5m), (F5) the proportion of the time the driver gazes at distant obstacles (beyond 5m), (F6) the proportion of the time the driver gazes at distant obstacles (beyond 5m), (F7) the proportion of the time the driver gazes at immovable obstacles, (F8) the proportion of the number of times the driver gazes at immovable obstacles, (F9) the proportion of the time the driver gazes at movable obstacles, (F10) the proportion of the number of times the driver gazes at movable obstacles, (F11) the average value of the driver's gazing distance, (F12) the maximum value of the driver's gazing distance, (F13) the minimum value of the driver's gazing distance, (F14) the median value of the driver's gazing distance, (F15) the variance of the driver's gazing distance, (F16) the average value of the driver's eye movement rate, (F17) the maximum value of the driver's eye movement rate, (F18) the median value of the driver's eye movement rate, (F19) the variance of the driver's eye movement rate, (F20) the number of saccades of the driver (the number of unconscious, sudden, rapid, and tiny movements of both eyes simultaneously when changing the fixation point), (F21) the saccade time (the time for the eyeball to move rapidly between two fixation points), (F22) the maximum value of the driver's saccade length (the distance between two fixation points), (F23) the minimum value of the driver's saccade length (the distance between two fixation points), (F24) the average value of the driver's saccade length (the distance between two fixation points), (F25) the gazing spatial density of the driver (referring to the distribution pixel density of fixation points within the road space obtained in S1), (F26) the total residence time / total running time (the gazing residence time on each obstacle / the running time of the system), (F27) the proportion of the driver's re - gaze (the number of times the eyes return from one fixation point to a previously gazed - at point, accounting for the proportion of the total saccade times).

[0175] Step S8: Transmit the eye movement features to the decision - making layer, and calculate the corresponding personality type of the driver through model operation in the decision - making layer.

[0176] In the embodiment of the present application, the vehicle transmits the above - mentioned twenty - seven groups of eye movement features to the decision - making layer, and calculates the corresponding personality classification of the driver through model operation in the decision - making layer.

[0177] Exemplarily, please refer to Figure 6 , Figure 6 which is the development flowchart of a personality recognition model provided by an embodiment of the present application. As Figure 6 shown, the development process of the personality recognition model includes Step S11, Step S12, Step S21, and Step S22, which are specifically as follows:

[0178] In the embodiments of the present application, the development process of the personality recognition model includes two parts: 1. Data collection part; 2. Model training part.

[0179] 1. Data collection part

[0180] Step S11: Recruit a certain number of driver volunteers with a male-female ratio close to 1:1. The volunteers fill out the Big Five Personality Questionnaire Scale to obtain the Big Five Personality scores, and classify them as high or low according to the average value. Those higher than the average value are recorded as "high", and those lower than the average value are recorded as "low".

[0181] Step S12: The volunteers are required to drive on a vehicle equipped with this device structure on a road with complex road conditions, and record the characteristic values of F1 - F27.

[0182] 2. Model training part

[0183] Step S21: Correlate all the characteristic values with the personality traits of the drivers one by one to form a data set. Randomly divide this data set, and divide the test set and the training set according to a ratio of 8:2.

[0184] Step S22: Use a variety of machine learning algorithms (including decision tree classifier, random forest classifier, SVM classifier, naive Bayes classifier, etc.), train on the test set, and then use the test set to comprehensively compare indicators such as F1 score, accuracy, recall rate, and precision, and select the best model as the model.

[0185] Exemplarily, please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a personality recognition device provided by an embodiment of the present application. Figure 7 The device shown is built into the vehicle to implement Figure 5 and Figure 6 the solutions shown.

[0186] As Figure 7 shown, the vehicle obtains obstacle information and driver's eye information through millimeter-wave radar, camera, and infrared camera, extracts features through the signal processing unit to obtain twenty-seven groups of characteristic values, and inputs the twenty-seven groups of characteristic values into the classifier to obtain the output of the classifier as the personality recognition result of the driver.

[0187] In the embodiments of the present application, the above solution identifies the eye movement information of the driver through an in-vehicle infrared camera, identifies the road information outside the vehicle through a millimeter-wave radar and a camera, and completes the identification of the driver by calculating the corresponding eigenvalues. Compared with the existing eye movement-based personality recognition technology, the above solution combines the specific road conditions and the driver's eye movement judgment, and can make full use of the driver's eye movement information to reflect the driver's personality traits. Compared with the existing in-vehicle voice, video and other personality recognition technologies, the above solution has less intrusion into the driver's privacy. The following beneficial effects can be achieved:

[0188] (1) In terms of privacy protection, only recording the road information and the driver's eye movement information can reduce the occurrence of privacy problems. (2) In terms of actual function, it can identify the driver's personality traits in a non-perceived environment. (3) In terms of scheme design, this solution takes into account two types of information, eye movement and actual road conditions, and can extract more eigenvalues to optimize the model for identifying personality through eye movement.

[0189] The above is only one embodiment of the present application and should not be regarded as a limitation of the present application. Professionals should understand that various deformations and modifications can be made to the embodiments to meet different application requirements. Therefore, the scope of the present application should be defined by the claims appended to the claims.

[0190] Please refer to Figure 8 which shows a block diagram of a classification device for driving scenarios provided by an exemplary embodiment of the present application. The classification device for driving scenarios can be implemented as all or part of a computer device in a hardware or a combination of hardware and software manner to implement all or part of the steps in the embodiments shown as above Figures 2 to 4 As shown Figure 8 The classification device for driving scenarios includes:

[0191] A data acquisition module 801, configured to acquire the eye video of the driver and the external environment data of the vehicle during the driving process of the vehicle;

[0192] An eye movement information determination module 802, configured to determine the eye movement information of the driver based on the eye video; the eye movement information includes the direction information of the driver's fixation point and the blinking information;

[0193] An obstacle information determination module 803, configured to determine the obstacle information around the vehicle based on the external environment data of the vehicle; the obstacle information includes the type of obstacles on the road in front of the vehicle and the distance between the obstacles and the vehicle; the obstacle type is a movable obstacle or an immovable obstacle;

[0194] An eye movement feature acquisition module 804, configured to acquire the eye movement features of the driver based on the eye movement information and the obstacle information of the driver;

[0195] The classification result acquisition module 805 is configured to input the eye movement features into a classification model and obtain the classification result output by the classification model; the classification result is used to indicate the classification type of the driver.

[0196] In some embodiments, the eye movement information determination module 802 is configured to input the eye video into an action recognition model and obtain the pupil information and blink information of the driver output by the action recognition model; the pupil information includes the position of the pupil; the blink information includes the blink frequency and the blink speed.

[0197] Based on the pupil information of the driver, determine the direction information of the driver's fixation point.

[0198] In some embodiments, the eye movement information determination module 802 is configured to obtain the first position and the second position of the driver's pupil within a first period of time; the first position is the position of the driver's pupil at a first moment; the second position is the position of the driver's pupil at a second moment; the first moment and the second moment are adjacent moments.

[0199] Obtain a first time difference, a second time difference, and an angle difference; the first time difference is the time difference between the duration of the vehicle's infrared camera emitting infrared rays to the first position and the duration of the infrared rays reflecting back from the first position to the infrared camera when the driver's pupil is at the first position; the second time difference is the time difference between the duration of the vehicle's infrared camera emitting infrared rays to the second position and the duration of the infrared rays reflecting back from the second position to the infrared camera when the driver's pupil is at the second position; the angle difference is the change angle between the infrared rays reflected back from the first position and the infrared rays reflected back from the second position received by the infrared camera.

[0200] Based on the first time difference, the second time difference, and the angle difference, determine the displacement distance of the driver's pupil.

[0201] Based on the displacement distance of the driver's pupil and the position of the driver's pupil, determine the displacement angle of the driver's pupil.

[0202] Through the displacement distance and the displacement angle, query and obtain the direction information of the driver's fixation point from the relationship between the displacement distance, the displacement angle, and the direction of the driver's fixation point.

[0203] In some embodiments, the eye movement information determination module 802 is configured to determine the displacement distance of the driver's pupil determined based on the first time difference, the second time difference, and the angle difference through the following formula 1 based on the first time difference, the second time difference, and the angle difference:

[0204] Formula 1:

[0205] Wherein, C represents the speed of light, T1 represents the first time difference, T2 represents the second time difference, and θ1 represents the angle difference.

[0206] In some embodiments, the obstacle information determination module 803 is configured to input the vehicle external environment data into an obstacle recognition model, and obtain the obstacle information around the vehicle output by the obstacle recognition model.

[0207] In some embodiments, the eye movement features include blink frequency, blink speed, close obstacle attention rate, far obstacle attention rate, close obstacle fixation time ratio, far obstacle fixation time ratio, small obstacle attention rate, small obstacle fixation time ratio, large obstacle attention rate, large obstacle fixation time ratio, movable obstacle fixation time ratio, movable obstacle attention rate, immovable obstacle attention rate, immovable obstacle fixation time ratio, average fixation distance, maximum fixation distance value, minimum fixation distance value, median fixation distance, fixation distance variance, average eye movement rate, maximum eye movement rate, median eye movement rate, eye movement rate variance, saccade count, saccade time, fixation spatial density, and total fixation duration.

[0208] Please refer to Figure 9 , Figure 9 FIG. is a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. The computer device 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read-only memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 further includes a basic input / output system (I / O system) 906 for facilitating the transfer of information between various devices within the computer, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.

[0209] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 such as a mouse and a keyboard for user input of information. The display 908 and the input device 909 are both connected to the central processing unit 901 through an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may further include an input / output controller 910 for receiving and processing inputs from multiple other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, a printer, or other types of output devices.

[0210] The mass storage device 907 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable medium provide non-volatile storage for the computer device 900. That is to say, the mass storage device 907 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0211] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cartridges, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art know that computer storage media is not limited to the above several. The above system memory 904 and mass storage device 907 can be collectively referred to as memory.

[0212] The computer device 900 can be connected to the Internet or other network devices through a network interface unit 911 connected to the system bus 905.

[0213] The memory also includes one or more programs. The one or more programs are stored in the memory, and the central processing unit 901 implements Figures 2 to 4 all or part of the steps in the method shown.

[0214] In an exemplary embodiment, a chip is further provided. The chip includes programmable logic circuits and / or program instructions, which are used to implement all or part of the steps of the methods shown in the above various embodiments of the present application when the chip runs on a computer device.

[0215] In an exemplary embodiment, a computer program product is further provided. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions to implement all or part of the steps of the methods shown in the above various embodiments of the present application.

[0216] In an exemplary embodiment, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement all or part of the steps of the methods shown in the above various embodiments of the present application.

[0217] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The above program can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disc, etc.

[0218] Those skilled in the art should be able to realize that in the above one or more examples, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer storage media and communication media, where the communication media includes any medium that facilitates the transfer of a computer program from one place to another. The storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0219] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A classification method for driving scenes, characterized in that: The method is performed by a vehicle, and comprises: During the driving process, the driver's eye video and external environment data are obtained; Based on the eye video, determining the eye movement information of the driver; the eye movement information includes the direction information of the driver's gaze point and the blink information; Determine obstacle information around the vehicle based on the vehicle external environment data; the obstacle information includes the type of obstacle on the road in front of the vehicle and the distance between the obstacle and the vehicle; the obstacle type is a movable obstacle or an immovable obstacle; Acquiring eye movement features of the driver based on the eye movement information of the driver and the obstacle information; The eye movement feature is input into a classification model to obtain a classification result output by the classification model, where the classification result is used to indicate the classification type of the driver.

2. The method according to claim 1, characterized in that The determining the eye movement information of the driver based on the eye video includes: Inputting the eye video into the action recognition model, obtaining the driver's pupil information and the blink information output by the action recognition model; the pupil information includes the position of the pupil; the blink information includes the blink frequency and the blink speed; Based on the pupil information of the driver, direction information of the driver's gaze point is determined.

3. The method according to claim 2, characterized in that The determining, based on the pupil information of the driver, the direction information of the driver's gaze point comprises: Acquire a first position and a second position of the driver's pupil within a first time; the first position is the position of the driver's pupil at a first moment; the second position is the position of the driver's pupil at a second moment; the first moment and the second moment are adjacent moments; Obtaining a first time difference, a second time difference, and an angle difference; the first time difference is the time difference between the time length of time that the infrared camera of the vehicle transmits infrared rays to the first position and the time length of time that the infrared rays are reflected from the first position back to the infrared camera when the pupil of the driver is in the first position; the second time difference is the time difference between the time length of time that the infrared camera of the vehicle transmits infrared rays to the second position and the time length of time that the infrared rays are reflected from the second position back to the infrared camera when the pupil of the driver is in the second position; the angle difference is the change angle between the infrared rays reflected from the first position and the infrared rays reflected from the second position received by the infrared camera; determining a displacement distance of the driver's pupil based on the first time difference, the second time difference, and the angle difference; Determining a displacement angle of the driver's pupil based on the displacement distance of the driver's pupil and the position of the driver's pupil; The direction information of the driver's gaze point is obtained by querying the relationship between the displacement distance, the displacement angle and the direction of the driver's gaze point through the displacement distance and the displacement angle.

4. The method according to claim 3, characterized in that The determining the displacement distance of the driver's pupil based on the first time difference, the second time difference and the angle difference includes: Based on the first time difference, the second time difference and the angle difference, the displacement distance of the driver's pupil determined based on the first time difference, the second time difference and the angle difference is determined by the following formula 1: Wherein, C represents the speed of light, T1 represents the first time difference, T2 represents the second time difference, and θ1 represents the angle difference.

5. The method according to claim 1, characterized in that The obtaining obstacle information around the vehicle based on the external environment data includes: The vehicle external environment data is input into an obstacle recognition model to obtain obstacle information around the vehicle output by the obstacle recognition model.

6. The method according to claim 1, characterized in that The eye movement features include blinking frequency, blinking speed, near-distance obstacle attention rate, far-distance obstacle attention rate, near-distance obstacle gaze time ratio, far-distance obstacle gaze time ratio, small obstacle attention rate, small obstacle gaze time ratio, large obstacle attention rate, large obstacle gaze time ratio, movable obstacle gaze time ratio, movable obstacle attention rate, immovable obstacle attention rate, immovable obstacle gaze time ratio, average gaze distance, maximum gaze distance value, minimum gaze distance value, median gaze distance, gaze distance variance, average eye movement rate, maximum eye movement rate, median eye movement rate, eye movement rate variance, number of saccades, saccade time, gaze space density and total gaze duration.

7. A classification device for driving scenes, characterized in that: The device comprises: A data acquisition module is used to acquire the driver's eye video and external environment data during vehicle driving; An eye movement information determination module, used to determine the driver's eye movement information based on the eye video; the eye movement information includes the direction information of the driver's gaze point and blink information; An obstacle information determination module, configured to determine obstacle information around the vehicle based on the vehicle external environment data; the obstacle information includes the type of obstacle on the road in front of the vehicle and the distance between the obstacle and the vehicle; the obstacle type is a movable obstacle or an immovable obstacle; An eye movement feature acquisition module, used for acquiring the eye movement features of the driver based on the eye movement information of the driver and the obstacle information; The classification result acquisition module is used to input the eye movement feature into the classification model to obtain the classification result output by the classification model; the classification result is used to indicate the classification type of the driver.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein instructions are stored in the memory, and the instructions are executed by the processor to implement the classification method for driving scenes according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores instructions, and the instructions are executed by a processor of a computer device to implement the classification method for driving scenes as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the classification method for driving scenes as described in any one of claims 1 to 6.

Images