A method and system for predicting driver attention focus based on multimodal and multi-scale feature fusion

Abstract

This invention provides a method and system for predicting driver attention focus based on multimodal and multi-scale feature fusion, comprising: multimodal data acquisition and preprocessing; using a multi-branch convolutional neural network (CNN) to extract multi-scale features from the multimodal data, then fusing the extracted features to output a multi-scale fused feature map; capturing spatial dependencies through a self-attention mechanism and embedding positional encoding to preserve geometric information; using ConVLSTM to model the fused features of temporally dynamic continuous frames as a temporal sequence as input, and using a coarse-grained prediction head to locate high-risk areas; using a fine-grained prediction head to output a pixel-level attention heatmap to accurately label the focus area; designing a loss function for training; and using the trained overall network model to achieve driver attention focus prediction. This invention balances accuracy and efficiency, significantly improving the generalization ability and practicality of the driver attention focus prediction model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road traffic safety, and in particular to a driver attention focus prediction method considering multi-modal and multi-scale feature fusion. BACKGROUND

[0002] With the rapid development of intelligent driving technology, accurately predicting the driver's attention focus has become one of the key technologies to improve the safety and reliability of the assisted driving system. Driver attention focus prediction aims to analyze the driver's visual behavior and vehicle operating state to infer the key road areas (such as pedestrians, obstacles, traffic signals, etc.) that the driver is paying attention to in real time, thereby providing key inputs for hazard warning, decision planning, and other modules. However, due to the complexity of driving scenarios and the dynamic nature of driver behavior, existing technologies still face many challenges in terms of prediction accuracy, robustness, and real-time performance.

[0003] Existing methods can be mainly divided into two categories: single-modal visual analysis-based methods and multi-modal data simple fusion-based methods. The former uses vehicle-mounted cameras to collect driver eye movement, head posture, or road scene images, and extracts visual features through deep learning models for prediction, but is easily affected by light, occlusion, and individual differences, and lacks correlation analysis of vehicle dynamic information, making it difficult to accurately distinguish attention focus. The latter fuses visual data and vehicle sensor data, and improves prediction effect through feature splicing or decision-level fusion, but often ignores the spatio-temporal correlation of multi-modal data, the feature fusion granularity is relatively coarse, the information utilization is not sufficient, and a single scale feature is often used, making it difficult to capture both local details and global semantics, resulting in prediction lag for long-distance small targets or sudden events. In addition, the driver's attention in the driving scene is influenced by the coupling of multi-scale spatio-temporal factors: in the short term, attention is affected by vehicle control actions (such as lane changing, braking) and nearby obstacles. Existing methods mostly use fixed time windows or single spatial resolution modeling, lack dynamic fusion mechanisms for multi-scale spatio-temporal features, and are difficult to adapt to rapid changes in complex driving scenarios. Moreover, traditional methods are not optimized for computational efficiency, making it difficult to meet the high-frame-rate real-time prediction requirements, limiting their application on embedded vehicle platforms. SUMMARY

[0004] In view of the problems existing in the prior art, there is an urgent need for a driver attention focus prediction method that can deeply fuse multi-modal data, adaptively capture multi-scale spatio-temporal features, and balance accuracy and efficiency, to improve the generalization ability and practicality of the prediction model in complex environments.

[0005] The technical solution of the present application is a driver attention focus prediction method based on multi-modal and multi-scale feature fusion, comprising the following steps:

[0006] Step 1, multi-modal data acquisition and preprocessing, the multi-modal data includes original RGB images, semantic segmentation images, depth images and optical flow images;

[0007] Step 2, using a multi-branch convolutional neural network (CNN) to extract multi-scale features from the original RGB images, semantic segmentation images, depth images and optical flow images respectively, and then fusing the extracted features to output a multi-scale fused feature map;

[0008] Step 3, taking the multi-scale fused feature map as input, capturing spatial dependency through self-attention mechanism and embedding position encoding to preserve geometric information; then stacking the fused features of consecutive frames as a time sequence through ConVLSTM modeling, positioning high-risk areas through a coarse-grained prediction head; outputting a pixel-level attention heat map through a fine-grained prediction head to accurately label the focus area;

[0009] Step 4, designing a loss function to train the overall network model consisting of steps 2-3;

[0010] Step 5, using the trained overall network model to realize driver attention focus prediction.

[0011] Further, in step 1, a pre-trained segmentation model is used to analyze the semantic categories in the road scene to obtain semantic segmentation images; the preprocessing includes image standardization, data augmentation and attention labeling, wherein the image standardization includes: histogram equalization of the RGB image; generating a multi-class label image from the semantic segmentation image; bilateral filtering denoising and linear normalization of the depth image; calculating dense optical flow based on FlowNet2.0, and scaling the motion vector to a certain range; data augmentation includes: randomly erasing a certain range of areas to simulate line-of-sight occlusion, adjusting brightness and contrast, and adding rain and fog noise;

[0012] Attention labeling includes: based on real driving data and eye tracker records, labeling the driver's attention focus area in the image and associating the risk level, wherein low risk corresponds to non-critical area, medium risk corresponds to potential risk area, and high risk corresponds to emergency area.

[0013] Further, ResNet-50 network is used to extract multi-scale features from the RGB image, PSPNet network is used to extract multi-scale semantic features from the semantic segmentation image, Hourglass network is used to extract multi-scale three-dimensional spatial features from the depth image, and 3D ResNet-18 network is used to extract multi-scale spatio-temporal motion features from the optical flow image.

[0014] Further, the calculation method of the multi-scale fused feature map is as follows:

[0015] F融合 =Sigmoid(W c RGB )⊙W d Depth ))+Softmax(F T Flow F Semantic

[0016] where F 融合 is the fused feature map, and W d is the channel attention weight matrix, obtained by the SE module, and is the matrix multiplication; the feature and the depth feature represent the RGB-depth feature association; and are the optical flow feature and the semantic feature, respectively, representing the motion-semantic association, Sigmoid is the activation function, and Softmax is the normalization exponential function;

[0017] When weighting the channel attention of the RGB feature and the depth image feature, first, the channel attention module is used to perform global pooling and weight calculation on the feature maps of the two, dynamically enhancing the channel weight containing spatial distance sensitive information, and fusing the weighted features;

[0018] For the motion information extracted from the optical flow network extracted from the optical flow image, the offset amount predicted by the optical flow is applied to the semantic feature through deformable convolution, dynamically adjusting its spatial sampling position to align the motion and semantic association.

[0019] Further, the coarse-grained prediction head includes a global average pooling layer, a fully connected layer, and a Softmax classification layer, and finally generates a scene-level probability distribution;

[0020] The fine-grained prediction head includes a deconvolution layer, a skip connection, and an activation function Sigmoid, and finally outputs an attention heat map.

[0021] Further, the designed loss function includes a weighted cross-entropy loss, a consistency loss, and a contrastive learning loss, and the final loss is the weighted sum of the three.

[0022] Further, the calculation formula of the weighted cross-entropy loss is as follows:

[0023]

[0024] where w , , corresponds to the non-critical area, w=2.0 corresponds to the potential risk area, w=4.0 corresponds to the high-risk area, and w=4.0 corresponds to the emergency area. . ​​​

[0025] Further, the calculation formula of consistency loss is:

[0026] = ||Downsample

[0027] Fine-grained feature map, obtained by coarse-grained prediction head and fine-grained prediction head respectively, and Downsample is a downsampling operation.

[0028] The calculation formula of contrast loss is:

[0029] =

[0030] RGB image deepest layer feature representation, depth image deepest layer feature representation, image deepest layer feature representation, .

[0031] Further, the training is divided into three stages:

[0032] In the first stage, data under sunny conditions in the daytime is used for training.

[0033] In the second stage, night data is added for training.

[0034] In the third stage, data under rain and fog weather conditions is added for training.

[0035] The application also provides a driver attention focus prediction system based on multi-modal multi-scale feature fusion, comprising:

[0036] One or more processors;

[0037] A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the driver attention focus prediction method based on multi-modal multi-scale feature fusion described in the above technical solutions.

[0038] The application proposes a driver attention focus prediction method considering multi-modal multi-scale feature fusion, and the advantages and beneficial effects mainly embody in the following aspects:

[0039] 1. Multi-modal data fusion

[0040] Cross-modal information integration: By synchronously collecting and processing multiple sensor data (such as RGB images, semantic segmentation maps, depth maps, and optical flow maps), the invention can comprehensively capture visual, spatial, and motion information in the driving environment. This multi-modal fusion strategy effectively improves the understanding ability of complex scenes.

[0041] Enhanced spatio-temporal correlation: Techniques such as deformable convolution are used to dynamically adjust sampling points, achieving spatial alignment of different modal data; at the same time, graph neural networks are used to model the topological relationship between road objects, strengthening the correlation between motion and semantics.

[0042] 2. Multi-scale feature extraction and fusion

[0043] Fine-grained to coarse-grained coverage: Using a cascading feature pyramid network (FPN), the invention performs layer-by-layer fusion of features of different modalities and scales, ensuring that the model can capture both local details (such as edges, textures) and understand global semantics (such as object categories, scene layout). This multi-level information representation enhances the model's response speed to long-distance small targets or sudden events.

[0044] Adaptive feature weighting: Through channel attention mechanisms and spatial attention mechanisms, the invention realizes automatic weighting of important feature regions, thereby improving the recognition accuracy of key information.

[0045] 3. Efficient spatio-temporal attention prediction model

[0046] Real-time performance optimization: By combining ConVLSTM units and time series modeling techniques, the invention can accurately capture the motion patterns of dynamic targets while maintaining a high frame rate. This makes real-time danger warning possible.

[0047] Fine-grained prediction: Not only can it generate scene-level attention distribution, but also output pixel-level attention heat maps, making the prediction results more accurate and helping to more accurately locate potential risk areas.

[0048] 4. Strong generalization ability and robustness

[0049] Staged training strategy: From sunny day to night to rainy and foggy weather, gradually introduce more challenging training data, so that the model has stronger environmental adaptability.

[0050] Loss function design: A combination of weighted cross-entropy loss, consistency loss, and contrastive loss is used, effectively improving the model's performance under different lighting conditions and enhancing the model's consistency learning for multi-modal data.

[0051] In summary, the application significantly improves the generalization ability and practicability of the driver attention focus prediction model by deeply fusing multi-modal data, adaptively capturing multi-scale spatio-temporal features, and considering both precision and efficiency. This innovation not only provides key technical support for intelligent driving assistance systems, but also lays a solid foundation for further exploring human-computer interaction in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0053] Figure 1 The implementation flowchart of the driver attention focus prediction method based on multi-modal multi-scale feature fusion provided by the application is shown in the figure.

[0054] Figure 2 The multi-modal multi-scale feature extraction design flowchart provided by the application is shown in the figure.

[0055] Figure 3 The multi-modal feature extraction fusion structure schematic diagram provided by the application is shown in the figure.

[0056] Figure 4 The driver attention focus prediction model schematic diagram provided by the application is shown in the figure. DETAILED DESCRIPTION

[0057] The technical solutions of the application will be further described in detail below with reference to the drawings.

[0058] As shown in Figure 1 and Figure 2 The driver attention focus prediction method based on multi-modal multi-scale feature fusion provided by the embodiments of the application includes the following steps:

[0059] Step 1, the original data of RGB images, semantic images, depth images and optical flow images are synchronously obtained by the vehicle-mounted RGB camera, ToF depth sensor, binocular camera, pre-trained segmentation model and FlowNet2.0 optical flow algorithm. Then the RGB image is converted to YUV color space and histogram equalization to enhance the contrast, the semantic segmentation image is labeled by DeepLabv3+, the depth image is denoised by bilateral filtering and linearly normalized to [0, 1], and the optical flow field is scaled to [-1, 1] range by motion vector. In addition, data augmentation techniques further improve the robustness of the model. The annotation information is based on real data recorded by an eye tracker, with an attention focus area labeled by an elliptical mask and associated with a risk level (low risk: road surface, non-moving obstacles; medium risk: distant pedestrians, stationary vehicles; high risk: suddenly appearing pedestrians, red light signals, etc.).

[0060] Step 2, the multi-modal feature extraction and fusion module is implemented, where the four modalities are input. The specific implementation process is that the RGB image is truncated to the branch of ResNet-50 conv4_x layer, extracting 14x14x1024 high-level semantic features (such as scene layout) and shallow edge texture details (such as conv2_x / 3_x layer); the semantic segmentation image generates 28x28x512 high-resolution semantic features through the pyramid pooling module of PSPNet; the depth image outputs 28x28x256 three-dimensional spatial features through the Hourglass network; the optical flow sequence processes 16 frames of data through 3D ResNet-18, generating 7x7x512 spatio-temporal motion features. Then the spatial sampling points of RGB and depth features are dynamically adjusted by deformable convolution to eliminate sensor bias, and the SE module (compression ratio r=16) is combined to weight the channels containing distance-sensitive information; the optical flow and semantic features are aligned in the spatial coordinate system through affine transformation (warp operation), and the channel and spatial attention are concatenated to focus on dynamic targets, such as shown in the figure. Figure 3 Further, the detected road objects (vehicles, pedestrians) are nodes of the graph neural network (GNN), and the edge relationship is constructed based on spatial distance and motion direction. The neighborhood features are aggregated by multi-layer GNN, and the risk heat map representing the interaction between objects is output, realizing the joint modeling of multi-modal features and topological relationships.

[0061] Among them, the fused features include spatial dimension, spatial consistency after alignment by deformable convolution and affine transformation. Semantic dimension, semantic label generated by PSPNet (such as lane line, pedestrian area). Motion dimension, spatio-temporal motion features extracted by 3D ResNet-18. Topological dimension: the interaction between objects coded by GNN (such as the proximity risk of vehicles and pedestrians), outputting multi-scale fused feature maps.

[0062] Step 3, the multi-scale fusion feature map is inputted to capture spatial dependencies through self-attention mechanism and embed position encoding to preserve geometric information; then, through ConVLSTM modeling, the fusion feature stack of continuous frames is inputted as a time sequence, and the global average pooling is used to generate a scene-level attention distribution to locate high-risk areas through a coarse-grained prediction head; through a fine-grained prediction head, the feature map is upsampled using a deconvolutional network to output a pixel-level attention heat map to accurately mark the focus area, as shown in Figure 4

[0063] Step 4, during the training process, weighted cross-entropy loss (high risk w=4.0, medium risk w=2.0, low risk w=1.0), consistency loss (constraint spatial alignment of coarse and fine-grained prediction) and contrast loss (align RGB and depth / optical flow feature space) are used, and the generalization is improved through a phased strategy: phase 1 (0-20 epochs) uses daytime sunny data (vehicle density <15 vehicles / frame) to establish basic recognition ability; phase 2 (21-40 epochs) introduces night data (ISO 1600 enhancement); phase 3 (41-60 epochs) adds rain and fog scenes (visibility <50m), gradually enhancing the model's robustness in complex environments. The final visual output includes scene-level risk area probability distribution, pixel-level heat map coordinates, etc., providing high-precision attention prediction support for driving assistance systems. Its advantages lie in multi-modal spatio-temporal correlation modeling, multi-scale feature adaptive fusion and real-time optimization, significantly improving prediction accuracy in complex scenarios and embedded platform applicability.

[0064] Step 5, use the trained overall network model to realize driver attention focus prediction.

[0065] Visualize the trained network model weight file as a heat map and overlay it on the original image to present the driver attention focus prediction effect.

[0066] The specific implementation of step 1, multi-modal data acquisition and preprocessing, is as follows:

[0067] Step 1.1, data source: synchronize multi-modal data collection through vehicle-mounted cameras, depth sensors, car recorders and traffic monitoring systems, including:

[0068] Raw RGB image: covers driver behavior inside the vehicle (such as head posture, hand movement) and road scene outside the vehicle (vehicles, pedestrians, traffic signs);

[0069] Semantic segmentation image: use a pre-trained segmentation model (such as DeepLabv3+) to analyze semantic categories (lane lines, obstacles, pedestrian areas) in the road scene;

[0070] ​Depth image: Acquire scene depth information through binocular camera or ToF sensor, construct three-dimensional space distance map;

[0071] Optical flow image: Calculate motion vector field based on consecutive frame RGB images, represent the motion trend of dynamic targets.

[0072] Sensor configuration: RGB camera: resolution 1920x1080@30fps, covering the driver inside the car (FOV 120°) and the scene outside the car (FOV 90°); ToF depth sensor: effective ranging 0.5-50m, accuracy ±2% @10m-baseline distance of binocular camera 12cm, generate 1280x720 depth map, use PTP protocol to achieve cross-device microsecond time synchronization, spatial alignment is completed by chessboard calibration method.

[0073] Step 1.2, data preprocessing:

[0074] Image standardization: RGB image (YUV color space conversion, histogram equalization); semantic segmentation map (generate 21-class label map using DeepLabv3+ (Backbone: Xception65)); depth map (bilateral filter denoising, linear normalization to [0,1]) and optical flow map (calculate dense optical flow based on FlowNet2.0, motion vector scaling to [-1,1]).

[0075] Data augmentation: Randomly erase 15-30% area to simulate line-of-sight occlusion, adjust brightness (±30%), contrast (±20%), add rain and fog noise.

[0076] Step 1.3, attention annotation: According to real driving data and eye tracker records, annotate the driver's attention focus area in the image (such as pedestrians, traffic lights), and associate the risk level (low / medium / high risk).

[0077] Low risk: Non-critical area (road surface, non-moving obstacle);

[0078] Medium risk: Potential risk area (such as distant pedestrians, stationary vehicles);

[0079] High risk: Emergency area (such as suddenly appearing pedestrians, red light signals, vehicles with sudden braking).

[0080] Data format: stored as JSON sequence, including frame number, focus coordinates, risk level, object ID association;

[0081] The specific implementation of multi-scale and multi-modal feature extraction and fusion in step 2 is as follows:

[0082] Step 2.1, Multi-modal feature extraction: A multi-branch convolutional neural network (CNN) is used to extract multi-scale features from RGB images, semantic images, and depth images. Low-level features: capture local details such as edges and textures through shallow convolutional layers. High-level features: extract global semantics such as scene categories and object relationships through deep convolutional layers. For optical flow images, 3D CNN is used to extract spatio-temporal motion features representing target displacement and velocity changes.

[0083] Low-level feature source: conv2_x or conv3_x layers from ResNet-50 (e.g., 7x7x64 or 14x14x256), capturing local details like edges and textures.

[0084] High-level feature source: conv4_x layer from ResNet-50 (14x14x1024), extracting global semantic information such as object categories and scene layout.

[0085] Table 1: Multi-branch convolutional neural network structure

[0086]

[0087] Step 2.2, Cross-modal multi-scale fusion: A cascaded feature pyramid network (FPN) is designed to fuse features from different modalities and scales layer by layer. For RGB and depth image features, channel attention weighting is applied. First, global pooling and weight calculation are performed on the features using a channel attention module to dynamically enhance channel weights containing spatial distance-sensitive information such as obstacles and road edges. The weighted features are then fused. Next, motion information is extracted using an optical flow network, and the predicted offset from the optical flow is applied to the semantic features using deformable convolution to dynamically adjust their spatial sampling positions for alignment with motion and semantic associations. Finally, detected road objects (vehicles, pedestrians) are used as nodes in a graph neural network (GNN), and edge relationships are constructed based on spatial distance and motion direction. The neighborhood features are aggregated through multiple layers of GNN to model topological interactions and output a risk heat map, achieving joint modeling of multi-modal features and topological relationships.

[0088] Step 3 is to establish a spatio-temporal attention focus prediction model, which is implemented as follows:

[0089] Spatio-temporal encoder:

[0090] The feature pyramid is used to fuse multi-scale information, and the multi-scale fused feature maps are input. The self-attention mechanism is used to capture spatial dependencies, and position encoding is embedded to preserve geometric information. ConVLSTM is used to model the temporal sequence of consecutive frame features, capturing the motion patterns of dynamic targets.

[0091] Multi-scale prediction head:

[0092] The feature map after fusion features is taken as an input of a space-time encoder, spatial dependence is captured through an attention mechanism, the fusion features of the continuous frames are stacked as a time sequence through ConVLSTM modeling time sequence dynamics, a coarse-grained prediction head is used to generate a scene-level attention distribution through global average pooling to locate high-risk areas, and a fine-grained prediction head is used to output a pixel-level attention heat map through a deconvolution network to accurately label the focus area.

[0093] Table 2 Multi-scale prediction head structure

[0094]

[0095] The specific implementation mode of joint optimization and training in step 4 is as follows:

[0096] Loss function design: weighted cross-entropy loss is used to strengthen the prediction accuracy of high-risk areas; multi-scale consistency loss: constraint the spatial consistency of coarse-grained and fine-grained prediction results; align the multi-modal feature space through contrastive learning to reduce the modal difference.

[0097] Weighted cross-entropy loss: ;

[0098] Wherein, , w = 2.0 Medium risk, w = 4.0 High risk, when calculating the weighted cross-entropy loss, the samples of high risk will be given higher weight, so as to pay more attention to the classification accuracy of these samples in the training process. ;

[0099] .

[0100] Consistency loss: = ||Downsample

[0101] Fine-grained feature map, , obtained through the coarse-grained prediction head and the fine-grained prediction head respectively; Downsample: down-sampling operation;

[0102] Contrastive loss: =

[0103] : feature representation of RGB image; : feature representation of depth image; feature representation of image; ;

[0104] Contrastive loss is used to measure the similarity and difference between RGB image features and other modal (depth, optical flow) features, helping the model learn cross-modal consistency.

[0105] Training strategy: pre-train each modal branch on independent tasks (such as semantic segmentation, optical flow estimation), and then progressive training. Stage 1 (0-20 epochs): sunny day scene, traffic density <15 vehicles / frame.

[0106] In the first stage, the model mainly uses data under sunny day conditions for training. These data usually have good lighting conditions and clear image quality, which is conducive to the model to learn basic features and patterns. The purpose is to initially establish the basic recognition ability of the model, and ensure that the model can accurately identify the target under good conditions.

[0107] Stage 2 (21-40 epochs): add night data (ISO increased to 1600)

[0108] In the second stage, the model begins to introduce night data for training. Night data usually has low lighting conditions, so the ISO setting needs to be increased to enhance image brightness. The purpose is to improve the recognition ability of the model under low light conditions, and ensure that the model can accurately identify the target at night.

[0109] Stage 3 (41-60 epochs): introduce rain and fog weather (simulation visibility <50m)

[0110] In the third stage, the model further introduces data under adverse weather conditions for training. These data simulate the low visibility conditions under rain and fog weather. The purpose is to enhance the robustness of the model under adverse weather conditions, and ensure that the model can accurately identify the target under low visibility conditions.

[0111] On the other hand, the embodiment of the present application also provides a driver attention focus prediction system based on multi-modal multi-scale feature fusion, comprising:

[0112] one or more processors;

[0113] a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the driver attention focus prediction method based on multi-modal multi-scale feature fusion described in the above technical solutions.

[0114] It should be understood that the above description is merely a detailed explanation of the preferred embodiments and is not intended to limit the patent protection scope of the present application. Any modification and alternation made by those skilled in the art without departing from the scope of the present application shall fall within the patent protection scope of the present application. The patent protection scope of the present application shall be subject to the appended claims.

Claims

1. A driver attention focus prediction method based on multi-modal multi-scale feature fusion, characterized in that, Comprising the following steps: Step 1, multi-modal data acquisition and preprocessing, the multi-modal data includes original RGB image, semantic segmentation image, depth image and optical flow image; Step 2, using multi-branch convolutional neural network CNN, respectively from the original RGB image, semantic segmentation image, depth image and optical flow image, multi-scale features are extracted, then the extracted features are fused, and multi-scale fusion feature map is outputted; The calculation method of multi-scale fusion feature map is as follows: F 融合 =Sigmoid(W c (F RGB )⊙W d (F Depth ))+Softmax(F T Flow F Semantic ) wherein F 融合 is the fused feature map, and W d is the channel attention weight matrix, obtained by the SE module, and is the matrix multiplication; the feature and the depth feature represent the RGB-depth feature association; and respectively, are the optical flow feature and the semantic feature, representing the motion-semantic association, Sigmoid is the activation function, and Softmax is the normalization exponential function. When the channel attention weighting is performed on the RGB feature and the depth image feature, firstly, the channel attention module is used to perform global pooling and weight calculation on the features of the two, the channel weight containing the spatial distance sensitive information is dynamically enhanced, and the weighted features are fused; For the motion information extracted from the optical flow image, the offset predicted by the optical flow is applied to the semantic feature through the deformable convolution, and the spatial sampling position is dynamically adjusted to align the motion and semantic correlation; Step 3, taking the multi-scale fusion feature map as input, the spatial dependence is captured through the self-attention mechanism, and the position encoding is embedded to retain the geometric information; then the fusion features of the continuous frames are stacked into time sequence as input through ConVLSTM modeling, the high-risk area is located through the coarse-grained prediction head; the pixel-level attention heat map is outputted through the fine-grained prediction head, and the focus area is accurately labeled; Step 4, a loss function is designed to train the overall network model constituted by steps 2-3; Step 5, the trained overall network model is used to realize the driver attention focus prediction.

2. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 1, characterized in that: In step 1, a pre-trained segmentation model is used to analyze the semantic categories in the road scene to obtain a semantic segmentation image; the preprocessing includes image annotation, data enhancement and attention annotation, wherein the image standardization includes: performing histogram equalization on the RGB image; generating a multi-class label image from the semantic segmentation image; performing bilateral filtering denoising and linear normalization on the depth image; calculating the dense optical flow of the optical flow image based on FlowNet2.0, and scaling the motion vector to a certain range; the data enhancement includes: randomly erasing a region within a certain range to simulate line of sight occlusion, adjusting brightness and contrast, and adding rain and fog noise; The attention annotation includes: according to the real driving data and the eye tracker record, the driver's attention focus area in the image is labeled and the risk level is associated, wherein the low risk corresponds to the non-key area, the medium risk corresponds to the potential risk area, and the high risk corresponds to the emergency area.

3. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 1, characterized in that: ResNet-50 network is used to extract multi-scale features in the RGB image, PSPNet network is used to extract multi-scale semantic features in the semantic segmentation image, Hourglass network is used to extract multi-scale three-dimensional spatial features in the depth image, and 3D ResNet-18 network is used to extract multi-scale space-time motion features in the optical flow image.

4. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 1, characterized in that: The coarse-grained prediction head includes a global average pooling layer, a fully connected layer and a Softmax classification layer, and finally generates a scene-level probability distribution; The fine-grained prediction head includes a deconvolution layer, a skip connection and an activation function Sigmoid, and finally outputs an attention heat map.

5. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 1, characterized in that: The designed loss function includes a weighted cross-entropy loss, a consistency loss, and a contrastive learning loss, and the final loss is a weighted sum of the three.

6. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 5, characterized in that: weighted cross-entropy loss The formula for calculating the weighted cross-entropy loss is: where w , , w = 2.0 for medium risk corresponding to non-critical areas, w = 4.0 for high risk corresponding to potential risk areas, and w = 6.0 for emergency areas; .

7. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 5, characterized in that: The formula for calculating the consistency loss is: = ||Downsample a fine-grained feature map, obtained by a coarse-grained prediction head and a fine-grained prediction head, respectively, Downsample is a down-sampling operation; The formula for calculating the contrastive loss is: = is the deepest feature representation for the RGB image, is the deepest feature representation for the depth image, is the deepest feature representation for the image, .

8. The driver attention focus prediction method based on multi-modal multi-scale feature fusion according to claim 1, characterized in that: The training is divided into three stages: In the first stage, data under sunny conditions during the day is used for training; In the second stage, night data is added for training; In the third stage, data under rainy and foggy weather conditions is added for training.

9. A driver attention focus prediction system based on multi-modal multi-scale feature fusion, characterized in that, Comprise: One or more processors; A storage device for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the driver attention focus prediction method based on multi-modal multi-scale feature fusion as claimed in any one of claims 1 to 8.

Images