Driver attention point rapid prediction method, system and application thereof

Abstract

The present application relates to a kind of driver attention visual point fast prediction method, system and its application, belong to visual point prediction technical field.First, driver visual field image is obtained, then driver visual field image is extracted using feature extraction model, and different scale first feature map and second feature map are obtained, feature extraction model uses MobileNetV3 network, finally, first feature map and second feature map are input, and driver attention visual point is predicted using attention visual point prediction model, to reduce interference information, avoid repeated processing, solve the problem that the accuracy of driver's attention visual point identification is poor, slow.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of view point prediction, in particular to a driver attention view point fast prediction method, system and application thereof. BACKGROUND

[0002] Since the birth of the automobile, improving the safety and comfort of vehicles has always been one of the important goals of vehicle development. In order to improve the safety and comfort of vehicles, technologies represented by advanced driver assistance systems (ADAS) are widely used, and the key promoting factor is the improvement of the perception ability of the vehicle to the surrounding environment. The improvement of the perception ability is accompanied by the use of a large number of sensors, in order to process the environmental information data generated by these sensors in real time, a powerful and energy-consuming computing unit needs to be equipped on the vehicle. Human drivers can quickly select the key part of the field of view, selectively allocate visual processing resources, and perceive the surrounding environment with only a small amount of energy consumption, and safely perform driving. Therefore, researching the human visual attention mechanism and establishing a driving behavior prediction model based on attention view point recognition plays an extremely important role in saving computing energy and improving the efficiency of machine vision environmental perception, and assisting vehicles or drivers to make faster and more accurate responses.

[0003] At present, most of the researches on human visual attention mechanism adopt the method of predicting the attention view point of human drivers based on color, edge and texture features. The limitation of this method is that the ability of feature extraction and information representation of the image is limited, there is much interference information, and usually the visual image data needs to be repeatedly processed, the data volume is large, the computational complexity is increased, and there are problems of poor accuracy and slow speed of the attention view point recognition of human drivers. SUMMARY

[0004] The purpose of the present application is to provide a driver attention view point fast prediction method, system and application thereof, which can quickly and accurately predict the attention view point of the driver.

[0005] In order to achieve the above purpose, the present application provides the following scheme:

[0006] A driver attention view point fast prediction method, the fast prediction method comprising:

[0007] obtaining a driver field of view image;

[0008] extracting features of the driver field of view image by using a feature extraction model to obtain first feature maps and second feature maps of different scales; the feature extraction model adopts a MobileNetV3 network;

[0009] using an attention view point prediction model to predict the driver attention view point by taking the first feature maps and the second feature maps as inputs.

[0010] A driver attention point rapid prediction system, the rapid prediction system comprising:

[0011] An image acquisition module is configured to acquire a driver field of view image;

[0012] A feature extraction module is configured to perform feature extraction on the driver field of view image by using a feature extraction model to obtain first feature maps and second feature maps of different scales; the feature extraction model adopts a MobileNetV3 network;

[0013] An attention point prediction module is configured to take the first feature maps and the second feature maps as input, and predict a driver attention point by using an attention point prediction model.

[0014] A driver steering wheel angle prediction method, the steering wheel angle prediction method comprising:

[0015] Acquiring a driver field of view image;

[0016] Performing feature extraction on the driver field of view image by using a feature extraction model to obtain first feature maps and second feature maps of different scales; the feature extraction model adopts a MobileNetV3 network;

[0017] Taking the first feature maps and the second feature maps as input, and predicting a driver attention point by using an attention point prediction model;

[0018] Taking the first feature maps and the second feature maps as input, and obtaining a semantic segmentation result by using a scene semantic segmentation model; the semantic segmentation result includes background, people, vehicles, and traffic signals;

[0019] Generating a first weight mask of the driver attention point and a second weight mask of the semantic segmentation result;

[0020] Performing weighted fusion on the first feature maps and the second feature maps by using the first weight mask and the second weight mask to obtain a fused feature map;

[0021] Taking the fused feature map as input, and predicting a steering wheel angle by using a steering wheel angle prediction model.

[0022] According to the specific embodiments of the present application, the following technical effects are disclosed:

[0023] The application is used for providing a driver attention visual point rapid prediction method, system and application thereof, first obtaining a driver visual field image, then using a feature extraction model to perform feature extraction on the driver visual field image to obtain first and second feature maps of different scales, the feature extraction model adopts a MobileNetV3 network, finally taking the first and second feature maps as inputs, and using an attention visual point prediction model to predict a driver attention visual point, so as to reduce interference information, avoid repeated processing, and solve the problems of low accuracy and slow speed of driver attention visual point recognition. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0025] Figure 1 The method flow chart of the rapid prediction method provided in embodiment 1 of the present application;

[0026] Figure 2 The structural schematic diagram of the feature extraction model provided in embodiment 1 of the present application;

[0027] Figure 3 The structural schematic diagram of the attention visual point prediction model provided in embodiment 1 of the present application;

[0028] Figure 4 The schematic diagram of the multi-task model framework provided in embodiment 1 of the present application;

[0029] Figure 5 The system block diagram of the rapid prediction system provided in embodiment 2 of the present application;

[0030] Figure 6 The method flow chart of the corner prediction method provided in embodiment 3 of the present application;

[0031] Figure 7 The principle block diagram of the corner prediction method provided in embodiment 3 of the present application;

[0032] Figure 8 The structural schematic diagram of the corner prediction model provided in embodiment 3 of the present application. DETAILED DESCRIPTION

[0033] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application. In order to make the above objectives, characteristics and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0034] Embodiment 1

[0035] The embodiment is used to provide a driver attention point rapid prediction method, as shown in Figure 1 The rapid prediction method comprises the following steps:

[0036] S1: obtaining a driver visual field image;

[0037] S2: performing feature extraction on the driver visual field image by using a feature extraction model to obtain first feature maps and second feature maps of different scales; the feature extraction model adopts a MobileNetV3 network;

[0038] In order to ensure the real-time performance and accuracy of the model, the feature extraction model uses a lightweight MobileNetV3 network as a backbone network, the structure of the backbone network is obtained through advanced network architecture search (NAS), and the backbone network has strong representation ability and can well extract the features of an RGB image. The model uses a depth-wise convolution (DC) method to construct a convolution layer, which means that each convolution kernel in the convolution layer only operates on a specific input channel, which can improve the rapidity of the model. The model uses a h-swish activation function to introduce nonlinearity into the model and improve the operation speed of the model, and the expression of the activation function is as follows:

[0039] ; (1)

[0040] In formula (1), x is the input of the hidden layer activation function of the model, which changes according to the change of the output of the previous layer network of the activation function layer.

[0041] The feature extraction model constructed in the embodiment is as shown in Figure 2 , and the network structure thereof is as shown in Table 1:

[0042] Table 1

[0043] Input Operation Output layer number Activation function Step size 288×512×3 conv2d, 3x3 16 h-swish 2 144×256×16 bneck, 3x3 16 ReLU6 1 144×256×16 bneck, 3x3 24 ReLU6 2 72×128×24 bneck, 3x3 24 ReLU6 1 72×128×24 bneck, 5x5 40 ReLU6 2 36×64×40 bneck, 5x5 40 ReLU6 1 36×64×40 bneck, 5x5 40 ReLU6 1 36×64×40 bneck, 3x3 80 h-swish 2 18×32×80 bneck, 3x3 80 h-swish 1 18×32×80 bneck, 3x3 80 h-swish 1 18×32×80 bneck, 3x3 80 h-swish 1 18×32×80 bneck, 3x3 112 h-swish 1 18×32×112 bneck, 3x3 112 h-swish 1

[0044] The driver's field of view image of the embodiment refers to the image in front of the vehicle within the driver's field of view, which is an RGB image. In order to adapt to different scales of information, the embodiment selects a multi-scale expression method when extracting image features, which can be understood as images with multiple resolutions. A series of multi-scale image features gradually decrease in resolution in a pyramid-like shape and are derived from the same original image, which is obtained by gradually downsampling. Different scales in this sampling process can be selected as image features to be used. The embodiment specifically selects two groups of features with feature sizes of 1 / 16 and 1 / 8 of the original image size. Then, taking the driver's field of view image as input, the feature extraction model can output two groups of features with different scales. Among them, the first feature map is a high-level feature, and its size can be 18x32, which is 1 / 16 of the original image size. The second feature map is a low-level feature, and its size can be 36x64, which is 1 / 8 of the original image size. The unit of the feature map size is pixel, which also indicates that the attention viewpoint prediction model can have high sensitivity to targets of different sizes. It should be noted that the size of the feature map is adjustable. The embodiment selects high-level features with a size of 1 / 16 and low-level features with a size of 1 / 8. Of course, other sizes of feature maps can also be selected as long as the sizes of the first feature map and the second feature map are different.

[0045] For the definition of high-level and low-level, the embodiment believes that the smaller the feature size, the lower the resolution, and the more inclined to express the global information of the original image. The higher the level of the feature, the larger the feature size, the higher the resolution, and the more inclined to express the local features of the original image. The lower the level of the feature.

[0046] S3: taking the first feature map and the second feature map as input, predicting the driver's attention viewpoint by using the attention viewpoint prediction model.

[0047] Before S3, the fast prediction method of the embodiment further includes training the attention viewpoint prediction model. The training process can include:

[0048] (1) obtaining a data set; the data set includes a plurality of training driver's field of view images, and a training first feature map, a training second feature map and a training driver's attention viewpoint corresponding to each training driver's field of view image;

[0049] The data set of the embodiment is established in two steps: first, collecting data in a real driving scene, including environment perception information and driver viewpoint attention information, wherein the environment perception information includes a driver field of view image, which can be collected by a vehicle-mounted camera, and the vehicle-mounted camera can be located at the center of the front windshield of the vehicle, so as to obtain a training driver field of view image; the driver viewpoint attention information is collected by an eye tracker worn by the driver. Then, a data preprocessing system is established, the videos of the eye tracker at the driver end and the vehicle-mounted camera at the vehicle-mounted end are all intercepted into pictures, and time stamp matching is performed, the viewpoint position based on the eye tracker coordinates is mapped to the vehicle-mounted end coordinates, so as to obtain a training driver attention viewpoint of each training driver field of view image. Taking the training driver field of view image as the input of the feature extraction model, the training first feature map and the training second feature map of each training driver field of view image can be obtained. For each training driver field of view image, taking the training first feature map and the training second feature map thereof as samples, and taking the training driver attention viewpoint thereof as a label, a data set can be constructed.

[0050] (2) constructing an initial prediction model;

[0051] The neural network layer of the attention viewpoint prediction model is set to construct an initial prediction model, and the complete network structure is as shown in Figure 3 The first feature map passes through 1 1x1 Conv (convolution layer) to obtain a first intermediate map; at the same time, the first feature map sequentially passes through Pooling (pooling layer), 2 1x1 Conv (convolution layer), Sigmoid activation function and Bilinear Upsample (bilinear upsampling) to obtain a second intermediate map; the first intermediate map and the second intermediate map are multiplied element by element, and then sequentially pass through 3 3x3 Conv (convolution layer), Pooling (pooling layer) and 1x1 Conv (convolution layer) to obtain a third intermediate map; the second feature map sequentially passes through 3 3x3 Conv (convolution layer), Pooling (pooling layer) and 1x1 Conv (convolution layer) to obtain a fourth intermediate map; the third intermediate map and the fourth intermediate map are added element by element, and then sequentially pass through 1 3x3 Conv (convolution layer) and Upsample (up-sampling) to obtain the driver attention viewpoint prediction value. In the embodiment, the output layer structure can be 288x512x1, and the size of the driver attention viewpoint prediction value is 288x512x1.

[0052] It should be noted that the driver attention viewpoint can be understood as a map annotated with the attention degree of the driver to each pixel point in the driver field of view image, and the attention degree can be embodied by a numerical value, for example, 1 is special attention, 10 is no attention, and the attention degree decreases from 1 to 10.

[0053] (3) training the initial prediction model using the dataset to obtain the attention point prediction model.

[0054] Specifically, the model parameter updating strategy of the embodiment can be: the model is trained using the training set and verified using the validation set, and each training and verification is regarded as completing a training cycle. In each training cycle, the loss of the current cycle of the model is calculated through the forward propagation link, and then the model parameters are updated using the Adam optimizer as the model optimizer in the backward propagation link. Based on this, (3) can include:

[0055] (3.1) dividing the dataset into a training set and a validation set;

[0056] (3.2) using the training set as the input of the initial prediction model, calculating the training loss value using the binary cross entropy loss function; updating the parameters of the initial prediction model using the Adam optimizer according to the training loss value to obtain an updated prediction model;

[0057] Using the training set as the input of the initial prediction model, the driver's attention point prediction value of each sample in the training set (i.e., the driver's field of view image for training) can be obtained. In the forward propagation link of model training, the embodiment selects the binary cross entropy (BCE) as the loss function of the attention point prediction model, and then uses the driver's attention point prediction value and the driver's attention point real value (i.e., the driver's attention point for training) of each sample in the training set as the input to calculate the training loss value using the binary cross entropy loss function.

[0058] The expression of the binary cross entropy loss function is as follows:

[0059] ; (2)

[0060] In formula (2), P gaze represents the mapping of the driver's attention point prediction value in the range of [0, 1] after being activated by the activation function; O gaze represents the driver's attention point prediction value.

[0061] ; (3)

[0062] In formula (3), Loss gaze represents the loss value of the driver's point prediction of the model; GT gaze represents the driver's attention point real value.

[0063] By substituting formula (2) into formula (3), the following can be obtained:

[0064] (4)

[0065] The loss value of each sample in the training set is calculated by using formula (4), and the sum of the loss values of all samples is obtained, that is, the training loss value.

[0066] The embodiment updates the parameters of the initial prediction model using the Adam optimizer, which means that the model uses the square of the gradient to adjust the learning rate and uses the momentum (Momentum) based on the sliding average of the gradient to adjust the gradient, so that the model can converge faster and more stably.

[0067] To prevent gradient explosion, before updating the parameters of the initial prediction model using the Adam optimizer, the rapid prediction method of the embodiment further comprises: setting the maximum gradient value of the Adam optimizer, specifically setting the maximum gradient value to 10, and setting the initial parameter value of the exponential decay rate of the momentum estimation in the optimizer to: β1=0.9; β2=0.999.

[0068] (3.3) Taking the validation set as the input of the updated prediction model, the validation loss value is calculated by using the binary cross-entropy loss function;

[0069] (3.4) Determine whether the validation loss values of the previous N consecutive iterations before the current iteration show a downward trend to obtain a first determination result;

[0070] (3.5) If the first determination result is no, end the iteration, and take the updated prediction model as the attention viewpoint prediction model;

[0071] The embodiment configures an early stopper for the deep learning model. In the periodic training process, if the validation loss value of the model on the validation set does not show a downward trend for N periods, the early stopper will forcibly end the training link, and the training process will end, avoiding more power consumption and highlighting lightweight. This setting can prevent the model from overfitting on the training set. In the embodiment, N can be 7.

[0072] (3.6) If the first determination result is yes, determine whether the preset iteration termination condition is reached;

[0073] The preset iteration termination condition can be reaching the maximum number of iterations or the validation loss value being less than a preset value.

[0074] (3.7) If yes, end the iteration, and take the updated prediction model as the attention viewpoint prediction model;

[0075] (3.8) If no, take the updated prediction model as the initial prediction model in the next iteration, and return to the step of "dividing the data set into a training set and a validation set".

[0076] To further improve the training efficiency of the model, the learning rate is further adjusted according to the loss value of the model on the validation set. Specifically, before judging whether the validation loss value of the previous N consecutive iterations shows a downward trend, the rapid prediction method of the embodiment further comprises: judging whether the validation loss value of the previous M consecutive iterations shows a downward trend; if not, the learning rate of the Adam optimizer in the next iteration process is lowered. That is, the embodiment configures the learning rate adjuster for the deep learning model. In the periodic training process, if the validation loss value of the model on the validation set does not show a downward trend for M cycles, the learning rate adjuster automatically reduces the value of the learning rate of the next cycle to one tenth of the current cycle, which means that the training link can adaptively adjust to the validation set to speed up the model training and updating process. In the embodiment, M can be 3, and the initial learning rate of the model can be set to: a =0.001.

[0077] After the attention point prediction model is trained, the first feature map and the second feature map extracted from the driver's field of view image by the lightweight neural network MobileNetV3 are input, and the attention point prediction model is used to predict the driver's attention point, without repeated data processing, which can greatly improve the speed and accuracy of attention point prediction.

[0078] As an optional implementation, after obtaining the first feature map and the second feature map of different scales, the rapid prediction method of the embodiment further comprises: taking the first feature map and the second feature map as input, and using a scene semantic segmentation model to obtain a semantic segmentation result, the semantic segmentation result including background, person, vehicle, and traffic signal. Based on this, the embodiment proposes a multi-task model framework for rapid prediction of driver's attention point and scene semantic segmentation, as shown in Figure 4 The framework takes the driver's field of view image as input, extracts multi-scale feature maps based on a lightweight deep network, and is used for predicting the driver's attention point and scene semantic segmentation.

[0079] The scene semantic segmentation model of the embodiment is used for Figure 3The illustrated attention viewpoint prediction model architecture is basically the same, and only the output layer structure needs to be adjusted on the basis of preserving the intermediate layer architecture. The output layer structure of the attention viewpoint prediction model is 288x512x1, and the effective output size is 36x64x1. The output layer structure of the scene semantic segmentation model of the environment in which the autonomous vehicle is located is 288x512x4, and the effective output size is 36x64x4. Moreover, the training process of the scene semantic segmentation model of this embodiment can also be the same as that of the attention viewpoint prediction model, and only the training driver attention viewpoint in the data set needs to be replaced by the training semantic segmentation result. The training semantic segmentation result can be the labeling of each training driver field of view image in the data set by a person. In the labeling process, the categories can be simplified into four categories: background, person, vehicle, and traffic signal. Scene segmentation refers to classifying each pixel of the current image to obtain the class to which it belongs. Because the model has four channels corresponding to background, person, vehicle, and traffic signal, each pixel point will finally be assigned a [0, 1] value under each channel, representing whether each pixel belongs to the class corresponding to the channel. The image segmentation result of the training driver field of view image is obtained through a basic image segmentation algorithm, and the semantics of different pixel points are defined by a person to label the four objects of background, person, vehicle, and traffic signal. Therefore, the training semantic segmentation result includes four, each of which is a pixel-level classification weight map, each pixel corresponds to a belonging category, and is used to identify whether each pixel of the training driver field of view image is background, person, vehicle, or traffic signal. If it is background, it can be marked as 1, and if it is not background, it can be marked as 0. Therefore, the training semantic segmentation result corresponding to the background can be obtained, and the training semantic segmentation results corresponding to the person, vehicle, and traffic signal can also be obtained in the same way.

[0080] Since the scene semantic segmentation model has four layers of neural networks, this embodiment selects the categorical cross entropy (Categorical Cross Entropy, CCE) as the loss function for semantic segmentation. The expression of the categorical cross entropy loss function is as follows:

[0081] (5)

[0082] (6)

[0083] In the above formula, P seg represents the mapping graph of the semantic segmentation result prediction value in the [0, 1] range after being activated by the activation function; O seg represents the semantic segmentation result prediction value; Loss seg represents the loss value of the semantic segmentation result prediction of the model; GTseg The semantic segmentation result true value (i.e., the semantic segmentation result for training) is represented.

[0084] It should be noted that the semantic segmentation result of the embodiment includes four, which are the semantic segmentation result corresponding to the background, the semantic segmentation result corresponding to the person, the semantic segmentation result corresponding to the vehicle, and the semantic segmentation result corresponding to the traffic signal. The semantic segmentation result refers to the result of whether each pixel point of the driver's field of view image belongs to the background, the person, the vehicle, or the traffic signal.

[0085] The first feature map and the second feature map are used to predict the driver's attention point and the semantic segmentation result, respectively, in the embodiment. The final prediction result is added to obtain the final prediction output.

[0086] In order to improve the efficiency of vehicle environmental perception, improve the speed of human driver attention point recognition, and improve the accuracy of driver behavior prediction, ensure that the control task is generated according to the corresponding important information, and improve the interpretability and stability of the model, the embodiment proposes a driver attention point fast prediction method based on deep learning. The information and point attention data in the driver's field of view are collected by using the vehicle-mounted camera and the head-mounted eye tracker. A multi-task learning strategy of directly inputting RGB images is adopted to extract key features. A multi-task model is trained by using a deep learning algorithm through the design of multiple loss functions. The human driver attention point can be identified and predicted, and the scene semantic segmentation result can be output.

[0087] Compared with the prior art, the advantages of the embodiment are as follows:

[0088] (1) Set special activation function and upper limit of optimizer gradient value, high network running efficiency

[0089] The depth separable convolution and h-swish activation function used in the driver attention point prediction model greatly improve the efficiency of the network while ensuring its nonlinearity. By setting the gradient upper limit in the running process of the neural network optimizer, the problem of gradient vanishing or gradient explosion in the neural network is alleviated, and the learning rate is increased to speed up the model convergence speed.

[0090] (2) Adaptive learning rate adjustment and early stopper setting, avoid overfitting, fast model convergence speed

[0091] The Adam optimizer is used in the driver attention point prediction model training process to adjust the learning rate by the square of the gradient, and the momentum based on the gradient is used to adjust the gradient, so the model can converge faster and more stably than the traditional stochastic gradient descent. At the same time, the learning rate adjuster and the early stopper are configured in the training process. The learning rate adjuster is set to reduce the learning rate to one tenth of the original when the model loss value on the validation set does not decrease for 3 consecutive periods during training. The early stopper is set to end the training when the model loss value on the validation set does not decrease for 7 consecutive periods during training, so as to prevent the model from overfitting on the training set.

[0092] Embodiment 2

[0093] The embodiment is used to provide a driver attention point fast prediction system, as shown in Figure 5 The fast prediction system comprises:

[0094] An image acquisition module M1 is configured to acquire a driver field of view image.

[0095] A feature extraction module M2 is configured to perform feature extraction on the driver field of view image by using a feature extraction model to obtain first feature maps and second feature maps of different scales; the feature extraction model adopts a MobileNetV3 network.

[0096] An attention point prediction module M3 is configured to take the first feature maps and the second feature maps as inputs, and predict a driver attention point by using an attention point prediction model.

[0097] Embodiment 3

[0098] At present, the end-to-end prediction model for driver behavior has poor interpretability, and the continuity of output control results such as accelerator opening degree and steering angle is poor. Due to the lack of important information screening mechanism in the scene, the model processes the input features indiscriminately, which is time-consuming and energy-consuming, and has the problem of poor practicability. Therefore, after predicting the driver attention point and the semantic segmentation result, the embodiment combines the semantic segmentation result and the predicted driver attention point to obtain a fusion feature map, and uses the fusion feature map as an input to train a steering angle prediction model, which can predict the driver steering wheel steering angle, so as to control the vehicle motion according to important information, and the practicability is better.

[0099] Specifically, based on the multi-task model framework of the established driver attention point prediction model and the scene semantic segmentation model, the driver attention point information and the scene semantic segmentation information can be obtained, which are used as a kind of weight mask, and a driver steering wheel steering angle prediction model is established according to the multi-task model design idea to realize the prediction of the steering wheel steering angle, that is, the embodiment is used to provide a driver steering wheel steering angle prediction method, as shown in Figure 6 andFigure 7 The corner prediction method comprises the following steps:

[0100] T1: obtaining a driver's field of view image;

[0101] T2: performing feature extraction on the driver's field of view image by using a feature extraction model to obtain first feature maps and second feature maps of different scales; the feature extraction model adopts a MobileNetV3 network;

[0102] T3: taking the first feature maps and the second feature maps as inputs, and predicting a driver's attention viewpoint by using an attention viewpoint prediction model;

[0103] T4: taking the first feature maps and the second feature maps as inputs, and obtaining a semantic segmentation result by using a scene semantic segmentation model; the semantic segmentation result comprises a background, a person, a vehicle and a traffic signal;

[0104] T5: generating a first weight mask of the driver's attention viewpoint and a second weight mask of the semantic segmentation result;

[0105] T5 can comprise:

[0106] (1) activating the driver's attention viewpoint by using a Sigmoid activation function to obtain a first weight mask of the driver's attention viewpoint;

[0107] The effective output size of the driver's attention viewpoint is 36x64x1, and the Sigmoid activation function is used to activate it to directly generate the first weight mask based on the driver's attention viewpoint.

[0108] (2) mapping the semantic segmentation result to a probability value by using a Softmax function, and performing weighted summation on the probability value to obtain a second weight mask of the semantic segmentation result.

[0109] The effective output size of the scene semantic segmentation result is 36x64x4, and the second weight mask is generated after merging the four output channels. The four classes corresponding to the four output channels are background, vehicle, person and traffic signal. First, the Softmax function is used to map the prediction results of the four output channels to probability values of the corresponding classes. Then, the probability values of the four output channels are weighted and summed according to the importance of each class to generate the second weight mask. The weights of the background, vehicle, person and traffic signal can be set to 0, 1, 1 and 1, respectively.

[0110] T6: performing weighted fusion on the first feature maps and the second feature maps by using the first weight mask and the second weight mask to obtain a fused feature map;

[0111] As Figure 2As shown, T6 can include: performing linear interpolation processing on the first feature map to obtain an interpolated feature map with the same size as the second feature map; merging the interpolated feature map and the second feature map to obtain a merged feature map; multiplying the merged feature map and the first weight mask to obtain a first weighted feature map; multiplying the merged feature map and the second weight mask to obtain a second weighted feature map; and merging the first weighted feature map, the second weighted feature map and the merged feature map to obtain a fused feature map. It should be noted that merging refers to a stacking process of two feature maps.

[0112] T7: taking the fused feature map as input, and predicting a steering wheel steering angle by using a corner prediction model.

[0113] As shown, Figure 8 the fused feature map is first reduced in size to 3x10x40 through 6 convolution operations, then the feature map is unfolded, and 4 fully connected layers are used to predict the steering angle of the driver's steering wheel. In order to avoid model overfitting, a Dropout operation is added between the fully connected layers.

[0114] When training the steering angle prediction model, the steering wheel steering angle for training can be collected from CAN bus data in a real driving scenario.

[0115] Compared with the prior art, the method of the embodiment has the advantages of multi-task model design and small model operation amount. This is because the input in the driver behavior prediction model follows the design idea of the multi-task model, directly performs weighted multiplication on two feature maps with different scales, can share the feature map extraction part with the previous driver viewpoint and semantic segmentation prediction, reduces the model operation amount, and improves the model efficiency.

[0116] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0117] The principles and implementation manners of the present application are described by using specific examples in this paper. The above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for rapid prediction of driver attention viewpoint, characterized in that, The fast prediction method includes: Acquire driver's field of vision image; The driver's field of vision image is used to extract features using a feature extraction model to obtain first and second feature maps at different scales; the feature extraction model uses the MobileNetV3 network. Using the first feature map and the second feature map as input, the driver's attention viewpoint is predicted using an attention viewpoint prediction model; Before predicting the driver's attentional viewpoint using the attentional viewpoint prediction model with the first feature map and the second feature map as input, the fast prediction method further includes: training the attentional viewpoint prediction model, specifically including: Obtain a dataset; the dataset includes multiple training driver field-of-view images and a training first feature map, a training second feature map, and a training driver attention viewpoint corresponding to each training driver field-of-view image; Construct an initial prediction model; The initial prediction model is trained using the dataset to obtain an attention viewpoint prediction model, specifically including: dividing the dataset into a training set and a validation set; using the training set as input to the initial prediction model and calculating the training loss value using the binary cross-entropy loss function; updating the parameters of the initial prediction model using the Adam optimizer based on the training loss value to obtain an updated prediction model; using the validation set as input to the updated prediction model and calculating the validation loss value using the binary cross-entropy loss function; determining whether the validation loss value of the previous N consecutive iterations shows a decreasing trend to obtain a first judgment result; if the first judgment result is negative, the iteration ends, and the updated prediction model is used as the attention viewpoint prediction model; if the first judgment result is positive, it is determined whether a preset iteration termination condition has been met; if yes, the iteration ends, and the updated prediction model is used as the attention viewpoint prediction model; if no, the updated prediction model is used as the initial prediction model in the next iteration, and the process returns to the step of "dividing the dataset into a training set and a validation set".

2. The method for rapid prediction of driver attention viewpoint according to claim 1, characterized in that, Before updating the parameters of the initial prediction model using the Adam optimizer, the fast prediction method further includes setting the maximum gradient value of the Adam optimizer.

3. The method for rapid prediction of driver attention viewpoint according to claim 1, characterized in that, Before determining whether the verification loss value of the previous N consecutive iterations shows a decreasing trend, the fast prediction method further includes: determining whether the verification loss value of the previous M consecutive iterations shows a decreasing trend; if not, then reducing the learning rate of the Adam optimizer in the next iteration.

4. The method for rapid prediction of driver attention viewpoint according to claim 1, characterized in that, After obtaining the first feature map and the second feature map at different scales, the fast prediction method further includes: using the first feature map and the second feature map as input, and using a scene semantic segmentation model to obtain semantic segmentation results; the semantic segmentation results include background, people, vehicles, and traffic signals.

5. A driver attention gaze point rapid prediction system, operating based on the rapid prediction method of claim 1, characterized in that, The rapid prediction system includes: Image acquisition module, used to acquire images of the driver's field of vision; The feature extraction module is used to extract features from the driver's field of vision image using a feature extraction model to obtain first feature maps and second feature maps at different scales; the feature extraction model uses the MobileNetV3 network. The attention viewpoint prediction module is used to predict the driver's attention viewpoint using the first feature map and the second feature map as input and the attention viewpoint prediction model.

6. A method for predicting driver steering wheel angle, characterized in that, The angle prediction method includes: Acquire driver's field of vision image; The driver's field of vision image is used to extract features using a feature extraction model to obtain first and second feature maps at different scales; the feature extraction model uses the MobileNetV3 network. Using the first feature map and the second feature map as input, the driver's attention viewpoint is predicted using an attention viewpoint prediction model; the attention viewpoint prediction model is the attention viewpoint prediction model as described in claim 1. Using the first feature map and the second feature map as input, a scene semantic segmentation model is used to obtain semantic segmentation results; the semantic segmentation results include background, people, vehicles, and traffic signals; Generate a first weight mask for the driver's attention viewpoint and a second weight mask for the semantic segmentation result; The first feature map and the second feature map are weighted and fused using the first weight mask and the second weight mask to obtain the fused feature map; Using the fused feature map as input, the steering wheel angle is predicted using the steering angle prediction model.

7. The method for predicting driver steering wheel angle according to claim 6, characterized in that, The first weight mask for generating the driver's attention viewpoint and the second weight mask for the semantic segmentation result specifically include: The driver's attention viewpoint is activated using the Sigmoid activation function to obtain the first weight mask of the driver's attention viewpoint; The semantic segmentation result is mapped to probability values ​​using the Softmax function, and the probability values ​​are weighted and summed to obtain the second weight mask of the semantic segmentation result.

8. The method for predicting driver steering wheel angle according to claim 6, characterized in that, The step of weighting and fusing the first feature map and the second feature map using the first weight mask and the second weight mask to obtain the fused feature map specifically includes: The first feature map is subjected to linear interpolation to obtain an interpolated feature map with the same size as the second feature map. The interpolated feature map and the second feature map are merged to obtain the merged feature map; The merged feature map is multiplied by the first weight mask to obtain the first weighted feature map; the merged feature map is multiplied by the second weight mask to obtain the second weighted feature map. The first weighted feature map, the second weighted feature map, and the merged feature map are merged to obtain the fused feature map.

Images