Hybrid Transform-based driver sight mapping method
By combining Transformer and ResNet18, combining binocular feature fusion module and cross attention network, the problem of line-of-sight mapping performance degradation caused by redundancy of facial image features in the prior art is solved, and high-precision driver line-of-view mapping to roads is achieved Estimation of gaze points in the environment improves the attention monitoring capabilities of the autonomous driving system.
Patent Information
- Application Number
- CN202410764384.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art has degraded performance in driver line-of-sight mapping due to redundancy in facial image features, and it is impossible to estimate the driver's concern in the road environment with high accuracy.
Using a hybrid Transformer-based driver's line of sight mapping method, face images and road images are collected through a dual-camera data acquisition system, facial and binocular features are extracted using Transformer and ResNet18, and feature fusion and cross attention network are performed through binocular feature fusion module and cross attention network. Regression prediction, output gaze coordinates.
High-precision driver's line of sight mapping to individual points on the road image is achieved, improving the accuracy and robustness of gaze estimation, and able to more effectively monitor the driver's attention state and reduce characteristic noise and information loss.
Smart Images

Figure CN118918401A_ABST
Abstract
Description
Technical Field
[0001] The invention studies the sight point of a driver in a road environment, and belongs to the technical fields of computer vision, image processing, human-computer interaction, etc. Background Art
[0002] Line of sight reflects human attention, and researchers can use line of sight to gain insight into human intentions and behaviors. Accurate line of sight estimation is an active research topic in the field of computer vision and has been widely used in driving tasks. Driving is a task that is highly dependent on vision. Drivers collect information related to driving tasks by looking at targets or areas of interest. Therefore, during driving, drivers need to maintain sufficient awareness to make correct decisions and perform a series of behavioral operations. With the rapid development of automation and intelligent technology, driver monitoring systems in advanced driver assistance systems (ADAS) play an increasingly important role in reducing traffic accidents caused by distracted driving. ADAS must monitor the driver's attention so that it can issue an alarm or take emergency measures to avoid traffic accidents when the driver is distracted. Driver attention assessment also plays an important role in the development of Level 3 autonomous driving systems. According to the automation levels established by the Society of Automotive Engineers (SAE), when a Level 3 autonomous driving system requests intervention, the driver should respond appropriately. Therefore, driver attention monitoring is particularly important for autonomous driving systems to assess whether the driver has sufficient awareness to take over control of the vehicle in complex or unpredictable driving scenarios.
[0003] The driver's line of sight is usually closely related to the driver's attention. At present, there are related studies that use the characteristics of the driver's eyes and head to monitor the driver's attention state. With the rapid development of eye tracking technology, appearance-based methods have been widely used in the task of driver gaze estimation. This method only needs to capture the driver's facial image through a traditional RGB camera, automatically extract facial features during the training process, and find a direct mapping function from image appearance to gaze direction or gaze position. With the rise of deep learning in the field of computer vision, researchers have begun to use CNN and Tranformer as appearance-based line of sight estimation methods.
[0004] (1) Deng Shulong. Driver's gaze area recognition and distraction detection method [P]. Hubei Province: CN202010334363.4, 2021-06-29. A driver's gaze area recognition and distraction detection method is proposed. The method estimates the head posture data through the driver's face image, and then determines the driver's gaze area with the gaze area mapping relationship. The method ultimately obtains the entire gaze area, but the method of the present invention is more specific and can map the driver's line of sight to a single point on the road image with high precision. It will inspire researchers to propose new solutions in the fields of line of sight mapping, driver intention recognition, driver attention prediction, human-vehicle interaction, etc., and further promote the development of autonomous driving systems.
[0005] (2) Wang Ke, Wang Jin, Cao Shuoyu. A deep learning method for continuous gaze estimation based on GRU [P]. Jiangsu Province: CN202311173058.1, 2024-05-28. A deep learning method for continuous gaze estimation based on GRU is proposed. The pre-trained ResNet-50 model is used to extract features from the input face image and perform feature dimensionality reduction, and the hidden state is input into the GRU for time series modeling, and finally the feature vector is mapped to a three-dimensional gaze vector. This method only uses face images as input, ignoring the intrinsic relationship between face and eyes at the feature granularity level. In fact, eye images provide fine-grained features focused on gaze, while face images provide coarse-grained features with richer information, because face images contain eye and head posture features, as well as features unrelated to gaze, such as background, skin color, and hair color. Therefore, developing more effective models is crucial to achieve better performance, robustness, and generalization in gaze estimation tasks. Summary of the invention
[0006] The purpose of the present invention is to provide a driver sight mapping method based on hybrid Transformer. It effectively solves the problem of sight mapping performance degradation caused by facial image feature redundancy, and achieves high-precision estimation of the driver's focus on the road environment during driving. The technical solution is as follows:
[0007] The objective of the present invention is achieved through the following technical solution: a driver sightline mapping method based on hybrid Transformer, which comprises the following steps:
[0008] (1) Build a dual-camera data acquisition system;
[0009] (2) Collecting driver’s facial image and road image;
[0010] (3) Preprocessing the driver's face image;
[0011] (4) The preprocessed images (including face images and binocular images) are input into the feature extraction network, where Transformer is used as the facial feature extractor and ResNet18 is used as the binocular feature extractor;
[0012] (5) The left and right eye features are fused through the binocular feature fusion module (BFFM);
[0013] (6) Introduce cross-attention networks at different scales. Specifically, let X1 be the binocular feature sequence after BFFM fusion, X2 be the feature sequence output by MSA (multi-head self-attention), and perform cross-attention on X1 and X2;
[0014] (7) Concatenate the Transformer and ResNet output features and globally average pool them;
[0015] (8) Input the pooled fusion features into KAN (Kolmogorov–Arnold Networks) for regression prediction and output the final gaze point coordinates X and Y;
[0016] (9) Construct an evaluation index and use the SmoothL1Loss loss function to measure the error between the predicted result and the true value, then backpropagate the error and update the network parameters.
[0017] Furthermore, in step (2), the front camera captures the driver's face image, and the scene camera captures the road image. The AprilTag is affixed to the gaze target in the actual scene. While ensuring driving safety, the driver focuses on the set tag during driving. The use of the tag can make it easier for the driver to concentrate, and during data marking, semi-automatic marking can be achieved by detecting the AprilTag, saving the time of manual data marking.
[0018] Furthermore, in step (3), the facial image and the left and right eye images are cropped using the Dlib method, and the driver's binocular images are enhanced using a UNet-based image enhancement model, and the UNet network architecture learns the nonlinear relationship between low-resolution and high-resolution images. The model consists of an encoder using a convolution block and a symmetric decoder using a deconvolution block, and ResNet18 is used as the backbone model of the encoder and decoder. In addition, a jump connection is added between the convolution block and the corresponding symmetric deconvolution block. And the Flickr-Faces-HQ (FFHQ) dataset is used for training.
[0019] Furthermore, the facial feature encoder in step (4) is composed of stacked downsampling modules (PatchEmbeding) and transformer blocks, where L represents the number of stacked layers. Its architecture is similar to the Swin transformer. For a face image of size H×W×3, convolution is used for downsampling operations, and the channel dimension is projected to C, and then goes through four stages of the transformer block. The eye feature encoder uses Resnet18 to extract binocular features. STAGE1 to STAGE4 are the four stages of Resnet18. During the training process, the feature extractor weights of the left eye and the right eye are kept shared.
[0020] Furthermore, in the BFFM in step (5), the left eye feature is represented by LF, and the right eye feature is represented by RF. BFFM uses a channel attention module (SE) to perform channel weighting on LF and RF respectively. Then, the Hadamard product is used to cross-multiply LF and RF, and then the features LF′ and RF′ are obtained through residual connection:
[0021] LF′=LF+SE(LF)⊙RF
[0022] RF′=RF+SE(RF)⊙LF
[0023] Where ⊙ represents the Hadamard product. In order to further improve the quality of binocular feature fusion, BFFM uses a spatial attention mechanism to enhance the global context information of the feature map. In this process, the fused LF′ and RF′ are spatially enhanced by SAM respectively. After SAM, a residual connection is established with the initial features, and then a dimensionality reduction operation is performed through convolution to obtain the features LF″ and RF″. Finally, the binocular fusion feature LRF is output by splicing LF″ and RF″:
[0024] LF″=LF+SAM(LF′)
[0025] RF″=RF+SAM(RF′)
[0026] LRF=Concat(LF″,RF″)
[0027] Furthermore, MCA is introduced in step (6). Different from self-attention, cross-attention calculates attention on two different sequences and is used to process the relationship between the two sequences.
[0028] Q=X1W q ,K=X2W k ,V=X2W v
[0029]
[0030] Furthermore, in step (7), the facial features and binocular features are concatenated and globally averaged pooled:
[0031] F fusion =Concat(F face ,F eye )
[0032] F = AdaptiveAvgPool(F fusion )
[0033] Furthermore, in step (8), KAN is inspired by the Kolmogorov-Arnold representation theorem. If f is a multivariate continuous function, then f can be written as a two-layer nested superposition of a finite number of single-variable continuous functions. The formula is as follows:
[0034]
[0035] where ψ q is the external function, φ q,p For internal functions.
[0036] X=cos(pitch)cos(yaw)
[0037] Y=cos(pitch)sin(yaw)
[0038] Furthermore, the index in step (9) is divided into 2D index and 3D index. The Euclidean distance between the real gaze point p(x1, y1) and the estimated point p′(x2, y2) is calculated as the 2D evaluation index, and the units are expressed in pixels and centimeters respectively:
[0039]
[0040] The 3D evaluation metric is represented by the angular difference between the true direction vector g and the estimated direction g′:
[0041]
[0042] Use the SmoothL1Loss loss function, which combines the advantages of square loss and absolute loss and is suitable for gaze estimation regression problems:
[0043]
[0044] Where x is the difference between the predicted value and the true value. When |x|≤1, the SmoothL1Loss loss function degenerates into a square loss; when |x|>1, it degenerates into an absolute loss.
[0045] Compared with the existing driving mapping method, the advantages of the present invention are:
[0046] (1) The device is simple and easy to operate. It can estimate the driver's gaze point in the road environment by only using a simple RGB camera to capture a face image.
[0047] (2) Although existing models can effectively predict the area inside the car where the driver is looking, our method is more specific and can map the driver's gaze to a single point on the road image with high accuracy.
[0048] (3) A gaze mapping network MSCA-Net with a novel architecture is proposed. The network extracts features in parallel through Tranformer and Resnet, takes binocular images and facial image vectors as network inputs respectively, introduces a cross-attention mechanism at different scales, and considers both local details and global semantic information of the features.
[0049] (4) A binocular feature fusion module (BFFM) is proposed, which can more effectively extract left and right eye features while reducing feature noise and information loss. By introducing an attention mechanism, BFFM can also better adapt to the imbalance of image regions and further improve the quality of fused feature representation, thereby improving the line of sight mapping accuracy of MSCA-Net. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 MSCA-Net structure diagram
[0051] Figure 2 Schematic diagram of dual-camera data acquisition system
[0052] Figure 3 AprilTag Marking Diagram
[0053] Figure 4 Transformer block diagram
[0054] Figure 5 Schematic diagram of BFFM structure DETAILED DESCRIPTION
[0055] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings.
[0056] A method for mapping a driver's sight line of sight based on a road environment comprises the following steps:
[0057] (1) Build a dual-camera data acquisition system;
[0058] (2) Collecting driver’s facial image and road image;
[0059] (3) Preprocessing the driver's face image;
[0060] (4) The preprocessed images (including face images and binocular images) are input into the feature extraction network, where Transformer is used as the facial feature extractor and ResNet18 is used as the binocular feature extractor;
[0061] (5) The left and right eye features are fused through the binocular feature fusion module (BFFM);
[0062] (6) Introduce cross-attention networks at different scales. Specifically, let X1 be the binocular feature sequence after BFFM fusion, X2 be the feature sequence output by MSA (multi-head self-attention), and perform cross-attention on X1 and X2;
[0063] (7) Concatenate the Transformer and ResNet output features and globally average pool them;
[0064] (8) Input the pooled fusion features into KAN (Kolmogorov–Arnold Networks) for regression prediction and output the final gaze point coordinates X and Y;
[0065] (9) Construct an evaluation index and use the SmoothL1Loss loss function to measure the error between the predicted result and the true value, then backpropagate the error and update the network parameters.
[0066] Furthermore, in step (2), the front camera captures the driver's face image, and the scene camera captures the road image. The dual-camera acquisition system is as follows: Figure 2 As shown in the figure, the AprilTag is attached to the gaze target in the actual scene. Figure 3 As shown, while ensuring driving safety, the driver looks at the set mark during driving. The use of this mark can make it easier for the driver to concentrate, and when marking data, semi-automatic marking can be achieved through the detection of AprilTag, saving time for manual data marking.
[0067] Furthermore, in step (3), the facial image and the left and right eye images are cropped using the Dlib method. Since the resolution of most of the eye areas in the image captured by the front camera is lower than 60*60 pixels, and the entire data acquisition process is dynamic, and the driver's body and head will move, resulting in motion blur in the image captured by the front camera, the UNet-based image enhancement model is used to enhance the driver's binocular images. The UNet network architecture learns the nonlinear relationship between low-resolution and high-resolution images. The model is composed of an encoder using a convolution block and a symmetric decoder using a deconvolution block, and ResNet18 is used as the backbone model of the encoder and decoder. In addition, Dense-Net is added between the convolution block and the corresponding symmetric deconvolution block. And the Flickr-Faces-HQ (FFHQ) dataset is used for training.
[0068] Furthermore, the facial feature encoder in step (4) is composed of stacked downsampling modules (PatchEmbeding) and transformer blocks, where L represents the number of stacked layers. Its architecture is similar to the Swin transformer. For a face image of size H×W×3, convolution is used for downsampling operations, and the channel dimension is projected to C, and then goes through four stages of the transformer block. The eye feature encoder uses Resnet18 to extract binocular features. STAGE1 to STAGE4 are the four stages of Resnet18. During the training process, the feature extractor weights of the left eye and the right eye are kept shared.
[0069] Furthermore, in the BFFM in step (5), the left eye feature is represented by LF, and the right eye feature is represented by RF. BFFM uses a channel attention module (SE) to perform channel weighting on LF and RF respectively. SE first performs a global average pooling operation on the input features of H×W×C, and the dimension becomes 1×1×C. Then, after passing through two fully connected layers, the Sigmoid activation function is used to normalize the values of each layer to the range of (0, 1) to represent the weight of each channel, and then the weight is multiplied by the input feature to obtain the channel enhanced feature. Then, the Hadamard product is used to cross-multiply LF and RF, and then the features LF′ and RF′ are obtained through residual connection:
[0070] LF′=LF+SE(LF)⊙RF
[0071] RF′=RF+SE(RF)⊙LF
[0072] Where ⊙ represents the Hadamard product. In order to further improve the quality of binocular feature fusion, BFFM uses a spatial attention mechanism to enhance the global context information of the feature map. In this process, the fused LF′ and RF′ are spatially enhanced by SAM respectively. SAM performs global maximum pooling and global average pooling on the input features of H×W×C in the channel dimension to obtain two feature maps of H×W×1. The results of global maximum pooling and global average pooling are spliced according to the channel, and the feature map size becomes H×W×2. Then, a 7×7 convolution operation is performed on the spliced result to obtain a feature map size of H×W×1. Then, the spatial attention weight matrix is obtained through the Sigmoid activation function, and the weight matrix is multiplied by the input feature to obtain the feature enhanced by spatial attention. After SAM, a residual connection is established with the initial feature, and then a dimensionality reduction operation is performed through convolution to obtain the features LF″ and RF″. Finally, the binocular fusion feature LRF is output by splicing LF″ and RF″:
[0073] LF″=LF+SAM(LF′)
[0074] RF″=RF+SAM(RF′)
[0075] LRF=Concat(LF″,RF″)
[0076] Furthermore, MCA is introduced in step (6). Different from self-attention, cross-attention calculates attention on two different sequences and is used to process the relationship between the two sequences.
[0077] Q=X1W q ,K=X2W k ,V=X2W v
[0078]
[0079] Furthermore, in step (7), the facial features and binocular features are concatenated and globally averaged pooled:
[0080] F fusion =Concat(F face ,F eye )
[0081] F = AdaptiveAvgPool(F fusion )
[0082] Furthermore, in step (8), KAN is inspired by the Kolmogorov-Arnold representation theorem. If f is a multivariate continuous function, then f can be written as a two-layer nested superposition of a finite number of single-variable continuous functions. The formula is as follows:
[0083]
[0084] where ψ q is the external function, φ q,p For internal functions.
[0085] X=cos(pitch)cos(yaw)
[0086] Y=cos(pitch)sin(yaw)
[0087] Furthermore, the index in step (9) is divided into 2D index and 3D index. The Euclidean distance between the real gaze point p(x1, y1) and the estimated point p′(x2, y2) is calculated as the 2D evaluation index, and the units are expressed in pixels and centimeters respectively:
[0088]
[0089] The 3D evaluation metric is represented by the angular difference between the true direction vector g and the estimated direction g′:
[0090]
[0091] Use the SmoothL1Loss loss function, which combines the advantages of square loss and absolute loss and is suitable for gaze estimation regression problems:
[0092]
[0093] Where x is the difference between the predicted value and the true value. When |x|≤1, the SmoothL1Loss loss function degenerates into a square loss; when |x|>1, it degenerates into an absolute loss.
[0094] Experiments have shown that the model MSCA-Net of the present invention has been trained on multiple datasets such as DGAZE, producing better results than the most advanced models. The network extracts features in parallel through Tranformer and Resnet, takes binocular images and facial image vectors as network inputs respectively, introduces cross-attention mechanisms at different scales, focuses on the eye area for effective feature extraction, and finally outputs the coordinates of the driver's gaze point. Although existing models can effectively predict the area inside the car where the driver is looking, the method in this paper is more specific and can map the driver's line of sight to a single point on the road image with high precision. There is still room for further improvement in the method in this paper, and future work will consider optimizing the detection speed of the network. It is believed that the method in this paper will inspire researchers to propose new solutions in the fields of line of sight mapping, driver intention recognition, driver attention prediction, human-vehicle interaction, etc., and further promote the development of autonomous driving systems.
[0095] It will be apparent to those skilled in the relevant art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the appended claims rather than the above description, and therefore it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.
Claims
1. A driver sightline mapping method based on hybrid Transformer, which is characterized by comprising the following steps: (1) Build a dual-camera data acquisition system; (2) Collecting driver’s facial image and road image; (3) Preprocessing the driver's face image; (4) The preprocessed images (including face images and binocular images) are input into the feature extraction network, where Transformer is used as the facial feature extractor and ResNet18 is used as the binocular feature extractor; (5) The left and right eye features are fused through the binocular feature fusion module (BFFM); (6) Introduce cross-attention networks at different scales. Specifically, let X1 be the binocular feature sequence after BFFM fusion, X2 be the feature sequence output by MSA (multi-head self-attention), and perform cross-attention on X1 and X2; (7) Concatenate the Transformer and ResNet output features and globally average pool them; (8) Input the pooled fusion features into KAN (Kolmogorov-Arnold Networks) for regression prediction and output the final gaze point coordinates X and Y; (9) Construct an evaluation index and use the SmoothLlLoss loss function to measure the error between the predicted result and the true value, then backpropagate the error and update the network parameters. According to claim 1, in step (2), the front camera is used to capture the driver's face image, and the scene camera is used to capture the road image. The dual-camera acquisition system is shown in FIG1. The AprilTag is attached to the gaze target in the actual scene. The AprilTag is shown in FIG2. While ensuring driving safety, the driver focuses on the set tag during driving. The use of the tag can make it easier for the driver to concentrate. In addition, during data marking, semi-automatic marking can be achieved by detecting the AprilTag, saving the time of manual data marking. According to claim 1, in step (3), the facial image and the left and right eye images are cropped using the Dlib method. Since the resolution of most of the eye areas in the image captured by the front camera is lower than 60*60 pixels, and the entire data acquisition process is dynamic, and the driver's body and head will move, resulting in motion blur in the image captured by the front camera, the UNet-based image enhancement model is used to enhance the driver's binocular images. The UNet network architecture learns the nonlinear relationship between low-resolution and high-resolution images. The model is composed of an encoder using a convolution block and a symmetric decoder using a deconvolution block, and ResNet18 is used as the backbone model of the encoder and decoder. In addition, Dense-Net is added between the convolution block and the corresponding symmetric deconvolution block. And the Flickr-Faces-HQ (FFHQ) dataset is used for training. According to claim 1, the facial feature encoder in step (4) is composed of stacked downsampling modules (PatchEmbeding) and transformer blocks, and L represents the number of stacked layers. Its architecture is similar to the Swin transformer. For a face image of size H×W×3, convolution is used for downsampling operations, and the channel dimension is projected to C, and then goes through four stages of the transformer block. The eye feature encoder uses Resnet18 to extract binocular features. STAGE1 to STAGE4 are the four stages of Resnet18. During the training process, the weights of the feature extractors of the left eye and the right eye are kept shared. According to claim 1, in step (5), the BFFM uses LF to represent the left eye feature and RF to represent the right eye feature. BFFM uses a channel attention module (SE) to perform channel weighting on LF and RF respectively. SE first performs a global average pooling operation on the input features of H×W×C, and the dimension becomes 1×1×C. Then, after passing through two fully connected layers, the Sigmoid activation function is used to normalize the values of each layer to the range of (0, 1) to represent the weight of each channel, and then the weight is multiplied by the input feature to obtain the channel enhanced feature. Then, the Hadamard product is used to cross-multiply LF and RF, and then the features LF′ and RF′ are obtained through residual connection: LF′=LF+SE(LF)⊙RF RF′=RF+SE(RF)⊙LF Where ⊙ represents the Hadamard product. In order to further improve the quality of binocular feature fusion, BFFM uses a spatial attention mechanism to enhance the global context information of the feature map. In this process, the fused LF′ and RF′ are spatially enhanced by SAM respectively. SAM performs global maximum pooling and global average pooling on the input features of H×W×C in the channel dimension to obtain two feature maps of H×W×1. The results of global maximum pooling and global average pooling are spliced according to the channel, and the feature map size becomes H×W×2. Then, a 7×7 convolution operation is performed on the spliced result to obtain a feature map size of H×W×1. Then, the spatial attention weight matrix is obtained through the Sigmoid activation function, and the weight matrix is multiplied by the input feature to obtain the feature enhanced by spatial attention. After SAM, a residual connection is established with the initial feature, and then a dimensionality reduction operation is performed through convolution to obtain the features LF″ and RF″. Finally, the binocular fusion feature LRF is output by splicing LF″ and RF″: LF″=LF+SAM(LF ′ ) RF″=RF+SAM(RF ′ ) LRF=Concat(LF″,RF″) According to claim 1, MCA is introduced in step (6). Different from self-attention, cross-attention calculates attention on two different sequences to process the relationship between the two sequences. Q=X1W q ,K=X2W k ,V=X2W v According to claim 1, in step (7), the facial features and binocular features are concatenated and globally averaged pooled: F fusion =Concat(F face ,F eye ) F = AdaptiveAvgPool(F fusion ) According to claim 1, in step (8), KAN is inspired by the Kolmogorov-Arnold representation theorem. If f is a multivariate continuous function, then f can be written as a two-layer nested superposition of a finite number of single-variable continuous functions. The formula is as follows: where ψ q is the external function, φ q,p For internal functions. X=cos(pitch)cos(yaw) Y=cos(pitch)sin(yaw) According to claim 1, the index in step (9) is divided into a 2D index and a 3D index, and the Euclidean distance between the real gaze point p(x1, y1) and the estimated point p′(x2, y2) is calculated as the 2D evaluation index, and the units are expressed in pixels and centimeters respectively: The 3D evaluation metric is represented by the angular difference between the true direction vector g and the estimated direction g′: Use the SmoothL1Loss loss function, which combines the advantages of square loss and absolute loss and is suitable for gaze estimation regression problems: in, x is the difference between the predicted value and the true value. When |x|≤1, the SmoothL1Loss loss function degenerates into a square loss; when |x|>1, it degenerates into an absolute loss.
