Driver visual distraction detection method and system fusing vehicle external traffic environment perception

Abstract

This invention discloses a method and system for detecting driver visual distraction by integrating perception of the external traffic environment. The method includes the following steps: constructing a driving scene dataset based on a panoramic map for feature extraction; designing a lightweight road semantic segmentation network; and introducing a visual large language model with deep logical reasoning capabilities to construct a visual distraction detection system that integrates perception of the external traffic environment. This invention uses an online visual large language model to analyze the semantic segmentation results of the external traffic environment and outputs in real time the target areas that the driver should focus on in the current scene. By verifying the consistency between this result and the driver's actual gaze point, accurate detection of the driver's visual distraction state is achieved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, and more specifically, relates to a method and system for detecting driver visual distraction by integrating perception of the external traffic environment. Background Technology

[0002] Driver visual distraction is a key human factor contributing to road traffic accidents. Statistics show that over 90% of information acquisition during driving decisions relies on the visual channel; therefore, real-time and accurate detection of driver visual distraction is crucial for improving driving safety. Currently, most mainstream methods for detecting driver visual distraction employ cascaded analysis strategies based on line-of-sight geometry. These methods primarily infer the driver's attention allocation by tracking eye gaze coordinates, statistically analyzing gaze duration and saccade amplitude, and other indicators. However, these solutions suffer from significant environmental semantic deficiencies: they often decouple driver gaze behavior from the dynamically changing external traffic environment, focusing only on eye kinematics while ignoring the semantic information of key risk targets (such as pedestrians, vehicles, and traffic signs) in the external traffic scene. This isolated analysis model makes it difficult for the system to establish an effective mapping relationship between gaze vectors and actual traffic risks. The limitations of existing methods are particularly pronounced in "false gaze" scenarios. A "false gaze" refers to a situation where the driver's physical gaze points ahead, but their visual attention is not focused on actual potential hazards. Due to the lack of perception and fusion of the external traffic environment context, it is easy to produce missed detections or false alarms, which seriously affects the reliability of visual distraction detection results.

[0003] Therefore, it is necessary to propose a driver visual distraction detection method and system that can deeply integrate real-time traffic environment perception. This will not only significantly improve the accuracy of advanced driver assistance systems in judging the driver's visual distraction state and enhance active safety performance, but also provide core technical support for quantitative evaluation of driving behavior and in-depth analysis of accident causes. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a method and system for detecting driver visual distraction that integrates perception of the external traffic environment. This significantly improves the accuracy, robustness, and interpretability of identifying driver visual distraction in complex traffic scenarios, providing key technical support for driving behavior evaluation and accident cause analysis.

[0005] To address at least one of the aforementioned technical problems, according to one aspect of the present invention, a method for detecting driver visual distraction by fusing perception of external traffic environment is provided, comprising the following steps:

[0006] A driving scene dataset based on panoramic maps was constructed for feature extraction;

[0007] A lightweight road semantic segmentation network is designed. By introducing the bidirectional segmentation network BiSeNetV2, the attention mechanism of the original BiSeNetV2 architecture is reconstructed in a targeted manner. By introducing spatial attention modules, efficient channel attention modules, and efficient multi-scale attention modules, a multi-dimensional collaborative enhancement segmentation model is constructed to achieve accurate mapping.

[0008] By introducing a visual large language model with deep logical reasoning capabilities, a visual distraction detection system that deeply integrates the external traffic environment is constructed.

[0009] Furthermore, in the construction of the driving scene dataset based on panoramic maps, assuming that the sphere and the cylinder are tangent to the equator, the latitude and longitude lines on the sphere are projected onto the cylinder, and then unfolded into a planar projection along a generatrix of the cylinder, thus unfolding the spherical data into a 2:1 strip-shaped planar image.

[0010] The geometric principle of the projection process is as follows: Let the three-dimensional coordinates of any point on the sphere be... Spherical coordinates are represented as azimuth angles. and polar angle The conversion relationship is as follows:

[0011] in Let be the radius of the sphere; map the spherical coordinates to the planar image coordinates. The formula is:

[0012] in and These represent the width and height of the panoramic image, respectively; the pixels on the sphere are precisely mapped to a planar coordinate system.

[0013] To meet the perception requirements of driving scenarios, the horizontal field of view directly in front of the vehicle is extracted from the original panoramic image. The core area; simultaneously, to eliminate interference from redundant information such as sparse sky above and vehicle body obstruction below, the vertical field of view was optimized and set to... The extraction process is as follows: Let the center viewpoint of the vehicle's forward direction be... The range of the forward-looking region in the spherical coordinate system is defined as follows:

[0014] By using inverse projection transformation, the viewpoint coordinates are mapped back to the panoramic coordinate system, thereby achieving accurate local feature extraction.

[0015] Furthermore, a spatial attention module is introduced at the output of the Detail Branch to generate a weight map with strong spatial discriminative power by explicitly modeling the spatial positional dependencies between pixels.

[0016] Given input feature map The computation process of the spatial attention module is mainly divided into two stages: spatial feature aggregation and spatial weight generation.

[0017] For the spatial feature aggregation stage, global average pooling and global max pooling are first performed along the channel dimension to aggregate the original features. 3D feature vectors are compressed into single-channel spatial descriptors;

[0018] For location Pixel at that location, average pooling operator Aggregation location The average response of all channels reflects the smoothness characteristics of the spatial region:

[0019] Meanwhile, the max pooling operator This tends to extract the strongest feature activation at that location, preserving the most discriminative texture information:

[0020] For the spatial weight generation stage, two spatial descriptors are generated. and It contains complementary structural information; the module concatenates the two along the channel dimension to obtain a dual-channel feature map that integrates multidimensional statistical information. ;

[0021] To establish long-range spatial constraints between pixels and eliminate local noise, the module applies a convolutional kernel with a size of [missing value]. Large kernel convolutional layer The spliced ​​features are fused; the output response is then processed. Represented as:

[0022] Activation function via Sigmoid Map features to The interval is used to generate the final spatial attention weight map. :

[0023] Spatial weights are applied to the original input feature map through element-wise multiplication. The above yields the feature output after boundary enhancement. :

[0024] .

[0025] Furthermore, the efficient channel attention module achieves adaptive local interactions between channels through one-dimensional convolution; for a given semantic feature map... The optimization process is divided into two stages: global information integration and adaptive local interaction.

[0026] For the global information integration phase, the efficient channel attention module first uses global average pooling to reduce the spatial dimensions within each channel. Compress the vector to generate a vector containing global context information. :

[0027] ;

[0028] For the adaptive local interaction phase, the size of the convolution kernel This determines the coverage of interactions between channels; to enable the model to adaptively adjust the interaction width according to network depth, a channel-dimensional mapping function is introduced:

[0029] Where the coefficient and Set as follows and Through linear transformation, the module can adapt to the number of channels. Increase the size of the convolution kernel Increase the receptive field;

[0030] Using convolution kernel size One-dimensional convolution For the aggregated vector The transformation is performed, and the weight coefficients for each channel are generated by combining the Sigmoid activation function. :

[0031] In this operation, the convolutional kernel slides along the channel dimension, and the weights at each output location are determined by that location and its adjacent channels (i.e., the channels covered by the convolutional kernel, each of size 1). The characteristics of the convolution kernel determine the overall structure of the system.

[0032] Ultimately, by using the weighting coefficients Applying this to the original feature map yields the calibrated semantic features:

[0033] .

[0034] Furthermore, the efficient multi-scale attention module establishes long-range spatial dependencies between pixels through grouped convolutions and cross-dimensional interactions; given the feature map fused from the BGA module... The internal logical evolution process mainly includes cross-space grouping and encoding, cross-dimensional interactive collaboration, and feature weighting and refinement output;

[0035] For the cross-spatial group coding stage, the module first inputs features Divided along the channel dimension The feature stream is divided into three non-overlapping subgroups to capture more discriminative grouped feature representations; then the feature stream enters three parallel computation branches: two 1x1 convolution branches and one 3x3 convolution branch.

[0036] In the 1x1 branch, the module employs coordinate attention logic, performing one-dimensional global pooling along both the horizontal and vertical directions to generate feature descriptors with orientation awareness:

[0037] This bidirectional encoding strategy can establish spatial relationships over long distances, providing a global perspective reference for the model;

[0038] For the cross-dimensional interactive collaboration stage, a cross-dimensional information interaction mechanism is introduced; the 3x3 convolutional branch is responsible for extracting local multi-scale feature distributions, while the direction vector generated by the 1x1 branch is transformed into spatial weights through the Softmax layer, guiding the network to focus on important structural regions.

[0039] The module compresses information from one branch using global average pooling and performs matrix multiplication with the original features of the other branch, thereby establishing a weight mapping between different feature scales; it also combines attention scores. Expressed as:

[0040] An organic link was established between local context and long-range spatial dependency;

[0041] In the feature weighting and refinement output stage, the module fuses the attention features generated from the three branches and applies the sigmoid activation function. Generate the final multi-scale weight map Weights are applied to the grouped features through element-wise multiplication, and the enhanced feature output is obtained after dimensionality reduction. :

[0042] .

[0043] Furthermore, the visual distraction detection fused with the large language model is specifically as follows: First, by transmitting real-time semantic segmentation results, the visual large language model is driven to deeply analyze the external scene and dynamically determine the important areas that the driver needs to focus on in the driving task; then, by combining the driver's gaze point detection results and verifying the consistency between the driver's gaze point area and the important areas, visual distraction determination is achieved.

[0044] According to another aspect of the present invention, a driver visual distraction detection system that integrates perception of the external traffic environment is provided to implement the above-described driver visual distraction detection method that integrates perception of the external traffic environment.

[0045] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the driver visual distraction detection method of the present invention that integrates perception of external traffic environment.

[0046] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the driver visual distraction detection method of the present invention that integrates perception of external traffic environment.

[0047] Compared with existing technologies, the beneficial effects of the above-described method of the present invention are as follows:

[0048] This invention is based on the construction of a driving dataset using panoramic maps. It utilizes panoramic maps with wide coverage and high update frequency as the data source, and constructs a dataset of real driving scenarios through image preprocessing, thereby reducing the cost of dataset construction and supporting the needs of long-term iteration and sustainable development.

[0049] This invention presents a lightweight road semantic segmentation network design, addressing the real-time and multi-source information fusion requirements of actual driving processes. It optimizes the BiSeNetV2 network by introducing a spatial attention module (SAM) to enhance edge detail perception, employing efficient channel attention (ECA) to optimize the response capability of semantic branches, and combining efficient multi-scale attention (EMA) to improve the accuracy of multi-target fusion. This enables accurate detection of key targets such as vehicles, pedestrians, and traffic signs, providing accurate traffic environment information for visual distraction detection.

[0050] This invention integrates visual distraction detection using a large language model. Based on the semantic segmentation results of the external traffic environment using an online visual large language model, it analyzes the data and outputs in real time the target areas that the driver should focus on in the current scene. By verifying the consistency of this result with the driver's actual gaze point, accurate detection of the driver's visual distraction state is achieved. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention.

[0052] Figure 1 This is a schematic diagram of panoramic map conversion according to a preferred embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the network architecture of a preferred embodiment of the present invention;

[0054] Figure 3 This is a flowchart of a preferred embodiment of the visual distraction detection method of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention.

[0056] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0057] Example 1:

[0058] like Figure 1-3 As shown, the present invention provides a driver visual distraction detection method that integrates external traffic environment perception, comprising the following steps:

[0059] 1. Construction of a driving scene dataset based on panoramic maps;

[0060] Panoramic maps are mostly stored using an equidistant cylindrical projection method, such as... Figure 1 As shown in (a), this projection method assumes that the sphere and the cylinder are tangent to the equator, projects the latitude and longitude lines on the sphere onto the cylinder, and then unfolds them into a plane along a generatrix of the cylinder, thus unfolding the spherical data into a 2:1 strip-shaped plane image.

[0061] The geometric principle of the projection process is as follows: Let the three-dimensional coordinates of any point on the sphere be... Its spherical coordinates are expressed as azimuth angles. and polar angle The conversion relationship is as follows:

[0062] in Let the radius be the sphere's radius. Map the spherical coordinates to the planar image coordinates. The formula is:

[0063] in and The width and height of the panoramic image are respectively (satisfying) Through the above transformation, the pixels on the sphere can be accurately mapped to the planar coordinate system, laying the geometric foundation for subsequent extraction of the vehicle's frontal view area and semantic segmentation.

[0064] To address the perception requirements of driving scenarios, this invention extracts the horizontal field of view (FOV) directly in front of the vehicle from the original panoramic image. The core area; simultaneously, to eliminate interference from redundant information such as sparse sky above and vehicle body obstruction below, the vertical field of view was optimized and set to... ,like Figure 1 As shown in (b).

[0065] The extraction process is as follows: Let the center viewpoint of the vehicle's forward direction be... The range of the forward-looking region in the spherical coordinate system is defined as follows:

[0066] By using inverse projection transformation, the viewpoint coordinates are mapped back to the panoramic coordinate system, thereby achieving accurate local feature extraction.

[0067] 2. Lightweight road semantic segmentation network design;

[0068] This invention introduces the bidirectional segmentation network BiSeNetV2 as its core architecture. Through its unique dual-branch design, this network ensures high-frequency real-time response while deeply considering both the detailed spatial information and high-level semantic context of the image, thus providing reliable pixel-level perceptual support for the accurate fusion of subsequent gaze points and environmental semantics. For example... Figure 2 As shown, to address the challenges of variable target scale and complex edge features in driving scenarios, this invention reconstructs the attention mechanism of the original BiSeNetV2 architecture. By introducing ECA (Efficient Channel Attention), SAM (Spatial Attention Module), and EMA (Efficient Multi-Scale Attention) modules, a multi-dimensional collaboratively enhanced segmentation model is constructed. While ensuring real-time performance, the optimized network significantly improves the semantic parsing ability of environmental entities in complex environments, laying the algorithmic foundation for accurate mapping of gaze points.

[0069] (1) Spatial Attention Module (SAM)

[0070] In real-world driving environments, drivers' gaze is often focused on extremely subtle environmental entities, such as distant traffic signs, lane line edges on the ground, or the outlines of pedestrians. However, lightweight networks, in order to maintain real-time inference speed, typically use continuous convolution strides for downsampling in detail branches, which inevitably leads to the loss of sub-pixel-level spatial details and edge blurring. Therefore, this invention introduces a Spatial Attention Module (SAM) at the output of the Detail Branch. This module generates a weight map with strong spatial discriminative power by explicitly modeling the spatial positional dependencies between pixels, thereby guiding the network to accurately lock onto key geometric structures and suppress useless background interference.

[0071] From a mathematical and structural logic perspective, the SAM mechanism no longer models the correlation between channels, but rather treats all channels as a whole, focusing on "where information has discriminative power in the spatial dimension." Given an input feature map... The module's calculation process is mainly divided into two stages: spatial feature aggregation and spatial weight generation.

[0072] For the spatial feature aggregation stage, in order to effectively capture the global feature distribution and salient edge feature points of the spatial region, the module first performs Global Average Pooling (GAP) and Global Max Pooling (GMP) along the channel axis. This operation will convert the original... The 3D feature vector is compressed into a single-channel spatial descriptor.

[0073] For location Pixel at that location, average pooling operator The average response of all channels at this location is aggregated, reflecting the smoothness characteristics of the spatial region:

[0074] Meanwhile, the max pooling operator This tends to extract the strongest feature activations (such as extreme points or edge signals) at that location, preserving the most discriminative texture information.

[0075] For the spatial weight generation stage, two spatial descriptors are generated. and It contains complementary structural information. The module then concatenates the two along the channel dimension to obtain a dual-channel feature map that integrates multidimensional statistical information. .

[0076] To establish long-range spatial constraints between pixels and eliminate local noise, the module applies a convolutional kernel with a size of [missing value]. Large kernel convolutional layer The spliced ​​features are then fused. During the design process, this invention compared and verified the convolution kernel size and tested standard... convolution, Performance of convolution and Mixed-SAM (Multi-Scale Convolution). Compared to regular small-sized convolutions, convolutional kernels with large receptive fields can capture a wider range of spatial background information, helping to delineate continuous and smooth object edges, and their output response... It can be represented as:

[0077] Finally, the Sigmoid activation function is used. Map features to The interval is used to generate the final spatial attention weight map. This weight map describes the importance of each pixel location in the image for the current segmentation task:

[0078] Finally, the spatial weights are applied to the original input feature map through element-wise product. The above yields the feature output after boundary enhancement. :

[0079] In a real-world vehicle environment, the introduction of the SAM module offers clear technical benefits: First, it establishes a spatial pixel priority mechanism, enabling the network to maintain geometric integrity for slender entities such as streetlights and guardrails even when faced with dynamic lighting changes or camera motion blur; second, The large-scale spatial prior provided by the convolution kernel effectively suppresses interfering noise such as reflections from car windows or shadows on the road surface. Compared to Convolutional or complex hybrid sampling strategies (MixSAM) are employed. The large kernel design exhibits greater robustness when dealing with long-span linear edges unique to driving scenarios, thereby significantly improving the model's accuracy in resolving edge details in complex driving environments.

[0080] (2) Efficient Channel Attention (ECA) module

[0081] After enhancing the spatial accuracy of the detail branches, another core challenge of the research lies in how to accurately extract and enhance semantic categories that are highly relevant to the driving task from the high-level abstract features of the semantic branches. In complex driving scenarios, different channels of the feature map often carry differentiated semantic weights (for example, some channels focus on recognizing the color attributes of traffic lights, while others are sensitive to road markings).

[0082] While the traditional squeeze-and-excitation (SE) mechanism can establish dependencies between channels, its dimensionality reduction operation through fully connected layers can easily disrupt the direct correspondence between channels and their weights and increase computational overhead. To address this issue, this invention introduces an efficient channel attention (ECA) module after the CEBlock in the semantic branch, aiming to improve the model's classification and recognition accuracy of key environmental entities through a cross-channel interaction strategy without dimensionality reduction.

[0083] From a mathematical and logical perspective, the ECA mechanism achieves adaptive local interaction between channels through one-dimensional convolution. For a given semantic feature map... The optimization process is divided into two stages: global information integration and adaptive local interaction.

[0084] In the global information integration phase, to obtain a channel-level global representation, the module first uses Global Average Pooling (GAP) to reduce the spatial dimensions within each channel. Compress the vector to generate a vector containing global context information. :

[0085] This operation condenses the feature distribution of each channel into a single scalar without introducing parameters, providing a statistical basis for subsequent channel weight allocation.

[0086] For the adaptive local interaction phase, in order to capture cross-channel dependencies while maintaining lightweight design, ECA employs one-dimensional convolutions instead of fully connected layers in traditional attention mechanisms. The size of the convolution kernel... This determines the coverage of inter-channel interactions. To enable the model to adaptively adjust the interaction width based on network depth, this invention introduces a channel-dimensional mapping function:

[0087] Where the coefficient and Set as follows and Through this linear transformation, the module can adapt to the increase in the number of channels. Increase the size of the convolution kernel This increases the receptive field, ensuring that deep networks have stronger cross-channel integration capabilities.

[0088] Subsequently, using a convolution kernel size of One-dimensional convolution For the aggregated vector The transformation is performed, and the weight coefficients for each channel are generated by combining the Sigmoid activation function. :

[0089] In this operation, the convolutional kernel slides along the channel dimension, and the weights at each output location are determined by that location and its adjacent channels (i.e., the channels covered by the convolutional kernel, each of size 1). The characteristics of the convolution kernel determine the computational complexity, and this local interaction mechanism effectively avoids the dramatic increase in computational complexity caused by global dependencies. Ultimately, by adjusting the weight coefficients... Applying this to the original feature map yields the calibrated semantic features:

[0090] In real-world vehicle analysis tasks, the introduction of the ECA module has a significant technical motivation: by precisely recalibrating channel responses, it significantly enhances the model's response strength to key categories such as "pedestrians," "traffic lights," and "moving targets" when faced with drastic changes in lighting or cluttered backgrounds (such as tree-lined roads or busy urban areas). Because this module avoids dimensionality compression, it more completely preserves the channel semantic information that is already scarce in lightweight networks. This "low-cost, high-performance" channel optimization scheme provides stable and pure semantic feature support for high-precision mapping of gaze points from physical space to cognitive semantics.

[0091] (3) Efficient Multi-Scale Attention (EMA)

[0092] After completing the spatial refinement of the detail branches and the channel recalibration of the semantic branches, the final key step in building a high-precision environmental foundation is to efficiently integrate heterogeneous features from both branches and eliminate feature misalignment during the fusion process. Addressing the characteristics of driving scenarios with vastly different target scales (e.g., wide lane lines in the foreground coexisting with tiny traffic signs in the background), this invention deploys an efficient multi-scale attention (EMA) module after the BGA fusion layer. This module aims to achieve deep feature collaboration and refinement through cross-spatial feature aggregation and a multi-scale parallel architecture.

[0093] The core design principle of the EMA module lies in breaking away from the single-scale limitations of traditional attention mechanisms. Through grouped convolutions and cross-dimensional interactions, it establishes long-range spatial dependencies between pixels. Given the feature map fused from the BGA module... Its internal logical evolution process mainly includes cross-space grouping and encoding, cross-dimensional interactive collaboration, and feature weighting and refinement output.

[0094] For the cross-spatial group coding stage, the module first inputs features Divided along the channel dimension The feature stream is then divided into non-overlapping subgroups to capture more discriminative grouped feature representations. Subsequently, the feature stream enters three parallel computation branches: two 1x1 convolutional branches (spatial orientation encoding) and one 3x3 convolutional branch (local scale enhancement).

[0095] In the 1x1 branch, to capture precise spatial location information, the module employs coordinate attention (CA) logic, performing one-dimensional global pooling along both the horizontal and vertical directions to generate feature descriptors with orientation awareness:

[0096] This bidirectional encoding strategy can establish spatial relationships over long distances, providing a global perspective reference for the model.

[0097] In the cross-dimensional interactive collaboration stage, EMA introduces a cross-dimensional information interaction mechanism to achieve deep alignment of multi-scale features. In this stage, the 3x3 convolutional branch is responsible for extracting local multi-scale feature distributions, while the direction vector generated by the 1x1 branch is transformed into spatial weights through the Softmax layer, guiding the network to focus on important structural regions.

[0098] The module compresses information from one branch using Global Average Pooling (GAP) and performs matrix multiplication with the original features of the other branch, thus establishing a weight mapping between different feature scales. Its combined attention score... This can be expressed as:

[0099] This interaction method effectively establishes an organic link between local context and long-range spatial dependencies.

[0100] In the feature weighting and refinement output stage, the module fuses the attention features generated from the three branches and applies the sigmoid activation function. Generate the final multi-scale weight map The weights are applied to the grouped features through element-wise multiplication, and the enhanced feature output is obtained after dimensionality reduction. :

[0101] In this invention, the application of the EMA module has a deep technical motivation: since drivers frequently switch their focus rapidly between different distances, EMA's multi-scale collaborative capability ensures that the model can automatically adjust its receptive field during the feature fusion stage. It acts like a "feature polisher," correcting the jagged edges and category ambiguity that may occur during the initial fusion of detail and semantic branches. By establishing long-range spatial dependencies, EMA significantly improves the model's semantic robustness in handling complex lighting scenarios, enabling the final pixel-level perception base to achieve feature consistency surpassing the original architecture while maintaining real-time performance.

[0102] 3. Visual distraction detection integrating large language models;

[0103] This invention introduces a Visual Large Language Model (LLM) with deep logical reasoning capabilities to construct a visual distraction detection method deeply integrated with the external traffic environment. First, by transmitting real-time semantic segmentation results, the LLM is driven to deeply analyze the external scene and dynamically determine the important areas that the driver needs to focus on during the driving task. Then, by combining the driver's gaze point detection results and verifying the consistency between the driver's gaze point area and the important areas, visual distraction detection is achieved. The process is as follows: Figure 3 As shown.

[0104] (1) Semantic information processing of traffic environment

[0105] First, the lightweight road semantic segmentation network described above is used to parse the video stream of the vehicle exterior scene frame by frame to generate a high-precision environmental semantic mask. Compared with the original image, the structured semantic labels effectively eliminate noise interference from low-level visual features such as lighting and texture, which not only provides a clean cognitive foundation for large models, but also significantly shortens the data transmission and processing time.

[0106] Secondly, based on the above semantic segmentation results, an input representation is constructed: the semantic mask, topological structure and target category information of the external traffic environment are structured and encoded to form a high-dimensional scene description vector, which serves as the input for logical reasoning of the large model.

[0107] Next, expert logic mapping is implemented: through prompt engineering, domain knowledge is embedded into the large model to simulate the cognitive logic of traffic safety experts. The large model not only passively identifies objects in the scene, but also actively combines its internalized complex road safety rules to dynamically allocate risk weights to various targets.

[0108] Finally, structured output constraints: To ensure efficient integration with the backend distraction detection algorithm, this invention imposes strict formatting constraints on the output of the large model. The model is forced to return only 1-3 key gaze regions that are most representative of the current scene.

[0109] (2) Visual distraction determination in asynchronous spatiotemporal alignment

[0110] Due to the physiological communication delay of approximately 1.2s to 1.5s in cloud-based large model inference, this invention proposes an asynchronous spatiotemporal alignment monitoring mechanism to address the mismatch between computation time and instantaneous visual behavior.

[0111] First, set the observation time window: set the detection cycle. This matches the model inference latency. Within this time window, the gaze detection module continuously records the driver's gaze coordinates at a high sampling rate (e.g., 30Hz), forming a dense sequence of gaze points.

[0112] Secondly, the semantic backtracking verification mechanism: when the large cloud model completes inference and returns the key gaze regions of the current scene. Then, the algorithm initiates a backtracking mechanism to retrieve all recorded driver gaze points within the corresponding time period. .

[0113] Then, the decision function model is as follows: Based on the backtracking results, the system determines the distraction state through the following logical function:

[0114] This function indicates that during the detection process, all of the driver's gaze points are located within the key gaze region. If it is internal, it is judged as focus ( Otherwise, it is judged as visual distraction. ).

[0115] Finally, a dynamic cyclic adaptive mechanism is implemented: periodic cyclic detection is performed according to the above process. This design ensures continuous coverage of macroscopic driving intentions while achieving an effective balance between detection accuracy and system real-time performance.

[0116] The above-described design method was used to conduct the verification.

[0117] 1. Data Collection

[0118] This invention captures high-definition panoramic images covering highways, urban arterial roads, overpasses, and complex intersections. After image preprocessing and projection transformation, the panoramic images are converted into planar images conforming to the viewpoint of the vehicle-mounted camera. A total of 3683 valid samples with good lighting conditions and clear target outlines are selected. The study performs refined semantic annotation on the samples, covering 19 key traffic participation elements, including vehicles, pedestrians, non-motorized vehicles, traffic signs, and lane lines. The dataset is divided into training and validation sets in an 8:2 ratio. This dataset contains the geometric features of real roads, providing high-quality supervision signals for model training.

[0119] 2. Training and Validation

[0120] To quantitatively evaluate the performance improvements of efficient channel attention (ECA), spatial attention module (SAM), and efficient multi-scale attention (EMA), this invention conducts ablation experiments using the basic BiSeNetV2 network as the baseline. The evaluation metrics selected are mean intersection-over-union ratio (mIoU) and... The F1 Score is the core evaluation indicator. Its calculation formula is as follows:

[0121] in, Number of categories; , , These represent the number of true positives, false negatives, and false positives, respectively. The experiment also differentiated four key configurations: single-scale testing (SS) to evaluate the real-time application performance of the vehicle-embedded device; single-scale flipping (SSC) to verify spatial symmetry robustness; and multi-scale (MSF) and multi-scale flipping (MSCF) tests to explore the upper limit of the model's accuracy in complex environments. The ablation experiment results for each module are shown in Table 1 (the data in the table are the average values ​​of 19 key traffic elements).

[0122] Experimental data show that each optimization module achieves differentiated gains in the performance of the baseline network through collaborative reconstruction of channel semantics, spatial boundaries, and multi-scale features. The ECA module steadily improves the model's basic accuracy under various configurations by adaptively adjusting channel weights; the SAM module, utilizing the spatial awareness capabilities of large-kernel convolution, performs excellently in multi-scale tests (MSCF), effectively enhancing the model's ability to capture slender and irregular geometric features in driving scenarios. Although the EMA module experiences slight fluctuations in single-scale metrics when used alone due to the lack of prior guidance, its feature alignment potential in multi-scale fusion (MSF / MSCF) allows it to significantly unlock the model's accuracy ceiling when combined with other modules. For example, in the +ECA+EMA configuration, its multi-scale flip F1 score reaches a maximum of 0.7197.

[0123] Table 1 Comparison of ablation experiment results for road semantic segmentation models

[0124]

[0125] When all three elements work together to form the full combined model (ECA+SAM+EMA), the model exhibits the strongest synergistic gain effect, especially in single-scale (ss) tests focused on vehicle deployment. With this configuration, the mIoU reaches 0.5973 (an improvement of approximately 1.34% over the baseline), and the F1 Score (ss), reflecting edge resolution accuracy, reaches a maximum of 0.7102. This result reveals a deep synergistic mechanism: ECA enhances the detection and recall of long-tailed small targets, SAM improves the accuracy of object boundary recognition, and EMA ensures the consistency of features at different distances. This multi-attention fusion strategy significantly enhances the system's robustness to semantic parsing in complex local driving scenarios while ensuring high-frequency real-time inference, providing a highly reliable environmental perception foundation for subsequent driver distraction determination based on a visual language model (VLM).

[0126] 3. Visual distraction detection

[0127] To verify the feasibility and practicality of the driver visual distraction detection method that integrates external vehicle scenes, this invention developed a complete detection system based on the PyQt framework. The system adopts a modular design, possesses good cross-platform compatibility, and can run stably on Windows and embedded Ubuntu systems. The NVIDIA Jetson Xavier NX embedded platform was selected as the test case.

[0128] The system comprises four core modules: a gaze detection module, which processes in-vehicle driver facial video streams in real time and outputs gaze classification results; an external scene semantic segmentation module, which performs real-time inference on road images and generates pixel-level semantic segmentation results; a visual large language model interface module, which designs a unified API communication protocol and supports dynamic switching of multiple mainstream VLM online services; and a visual distraction determination module, which, based on the key target recognition results of VLM, compares the driver's gaze points within the detection period to complete visual distraction detection.

[0129] In the test cases of this invention, the Qwen-VL-Plus model was used as the inference engine, and specialized prompts for driving scenarios were designed to guide the model in identifying key targets. The prompts are designed as follows:

[0130] "You are a professional road traffic safety expert. Please analyze this road semantic segmentation image."

[0131] This image is the semantic segmentation result of a single pixel value, where each pixel value corresponds to one of the following categories:

[0132] 0: road, 1: sidewalk, 2: building, 3: wall, 4: fence,

[0133] 5: pole, 6: traffic light, 7: traffic sign, 8: vegetation, 9: terrain,

[0134] 10: sky, 11: person, 12: rider, 13: car, 14: truck,

[0135] 15: bus, 16: train, 17: motorcycle, 18: bicycle

[0136] The actual category IDs present in the image: {result_dict['image_info']['unique_categories']}

[0137] Please divide the image into 6 rows and 11 columns, and number the divided areas starting from 1 in order from left to right and top to bottom; based on professional knowledge of road traffic safety, determine the 1-3 areas that the driver needs to pay the most attention to in the current driving scenario, and sort them from high to low attention.

[0138] Please return only the area codes, separated by commas, without any other text descriptions.

[0139] The system software interface adopts a multi-view design, simultaneously displaying the original video stream, gaze point recognition results, semantic segmentation status, and visual distraction judgment results, providing intuitive visualization support for testing and verification.

[0140] Through actual deployment and testing on this embedded platform, this solution verifies the feasibility of a visual distraction detection system that integrates vehicle-exterior scene semantics and VLM advanced inference in a resource-constrained environment, laying a solid technical foundation for subsequent product applications.

[0141] Example 2:

[0142] The computer-readable storage medium of this embodiment stores a computer program that, when executed by a processor, implements the steps in the driver visual distraction detection method fused with external traffic environment perception of Embodiment 1.

[0143] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0144] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0145] Example 3:

[0146] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the driver visual distraction detection method that integrates external traffic environment perception of Embodiment 1.

[0147] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0148] Those skilled in the art will understand that the content disclosed in the embodiments can be provided as a method, system, or computer program product. Therefore, this solution can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this solution can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0149] This solution is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of this solution. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0153] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

[0154] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A method for detecting driver visual distraction by integrating perception of the external traffic environment, characterized in that, Includes the following steps: A driving scene dataset based on panoramic maps was constructed for feature extraction; A lightweight road semantic segmentation network is designed. By introducing the bidirectional segmentation network BiSeNetV2, the attention mechanism of the original BiSeNetV2 architecture is reconstructed in a targeted manner. By introducing spatial attention modules, efficient channel attention modules, and efficient multi-scale attention modules, a multi-dimensional collaborative enhancement segmentation model is constructed to achieve accurate mapping. Visual distraction detection is achieved by integrating a large visual language model and introducing a visual large language model with deep logical reasoning capabilities to construct a visual distraction detection system that deeply integrates the external traffic environment.

2. The method as described in claim 1, characterized in that, In the construction of the driving scene dataset based on panoramic maps, it is assumed that the sphere and the cylinder are tangent to the equator. The latitude and longitude lines on the sphere are projected onto the cylinder, and then the projection is unfolded into a plane along a generatrix of the cylinder, thus unfolding the spherical data into a 2:1 strip-shaped planar image. The geometric principle of the projection process is as follows: Let the three-dimensional coordinates of any point on the sphere be... Spherical coordinates are represented as azimuth angles. and polar angle The conversion relationship is as follows: , in Let be the radius of the sphere; map the spherical coordinates to the planar image coordinates. The formula is: , in and These represent the width and height of the panoramic image, respectively; the pixels on the sphere are precisely mapped to a planar coordinate system. To meet the perception requirements of driving scenarios, the horizontal field of view directly in front of the vehicle is extracted from the original panoramic image. The core area; simultaneously, to eliminate interference from redundant information such as sparse sky above and vehicle body obstruction below, the vertical field of view was optimized and set to... The extraction process is as follows: Let the center viewpoint of the vehicle's forward direction be... The range of the forward-looking region in the spherical coordinate system is defined as follows: , By using inverse projection transformation, the viewpoint coordinates are mapped back to the panoramic coordinate system, thereby achieving accurate local feature extraction.

3. The method as described in claim 2, characterized in that, A spatial attention module is introduced at the output of the Detail Branch to generate a weight map with strong spatial discriminative power by explicitly modeling the spatial positional dependencies between pixels. Given input feature map The computation process of the spatial attention module is mainly divided into two stages: spatial feature aggregation and spatial weight generation. For the spatial feature aggregation stage, global average pooling and global max pooling are first performed along the channel dimension to aggregate the original features. 3D feature vectors are compressed into single-channel spatial descriptors; For location Pixel at that location, average pooling operator Aggregation location The average response of all channels reflects the smoothness characteristics of the spatial region: , Meanwhile, the max pooling operator This tends to extract the strongest feature activation at that location, preserving the most discriminative texture information: , For the spatial weight generation stage, two spatial descriptors are generated. and It contains complementary structural information; the module concatenates the two along the channel dimension to obtain a dual-channel feature map that integrates multidimensional statistical information. ; To establish long-range spatial constraints between pixels and eliminate local noise, the module applies a convolutional kernel with a size of [missing value]. Large kernel convolutional layer The spliced ​​features are fused; the output response is then processed. Represented as: , Activation function via Sigmoid Map features to The interval is used to generate the final spatial attention weight map. : , Spatial weights are applied to the original input feature map through element-wise multiplication. The above yields the feature output after boundary enhancement. : 。 4. The method as described in claim 3, characterized in that, The efficient channel attention module achieves adaptive local interactions between channels through one-dimensional convolution; for a given semantic feature map... The optimization process is divided into two stages: global information integration and adaptive local interaction. For the global information integration phase, the efficient channel attention module first uses global average pooling to reduce the spatial dimensions within each channel. Compress the vector to generate a vector containing global context information. : ； For the adaptive local interaction phase, the size of the convolution kernel This determines the coverage of interactions between channels; to enable the model to adaptively adjust the interaction width according to network depth, a channel-dimensional mapping function is introduced: , Where the coefficient and Set as follows and Through linear transformation, the module can adapt to the number of channels. Increase the size of the convolution kernel Increase the receptive field; Using convolution kernel size One-dimensional convolution For the aggregated vector The transformation is performed, and the weight coefficients for each channel are generated by combining the Sigmoid activation function. : , The convolutional kernel slides along the channel dimension, and the weight of each output position is determined by the features of the output position and its adjacent channels. Ultimately, by using the weighting coefficients Applying this to the original feature map yields the calibrated semantic features: 。 5. The method as described in claim 4, characterized in that, An efficient multi-scale attention module establishes long-range spatial dependencies between pixels through grouped convolutions and cross-dimensional interactions; given the feature map fused from the BGA module... The internal logical evolution process mainly includes cross-space grouping and encoding, cross-dimensional interactive collaboration, and feature weighting and refinement output; For the cross-spatial group coding stage, the module first inputs features Divided along the channel dimension Multiple non-overlapping subgroups are used to capture more discriminative grouping feature representations; subsequently The feature flow enters three parallel computation branches: two 1x1 convolution branches and one 3x3 convolution branch; In the 1x1 branch, the module employs coordinate attention logic, performing one-dimensional global pooling along both the horizontal and vertical directions to generate feature descriptors with orientation awareness: , This bidirectional encoding strategy can establish spatial relationships over long distances, providing a global perspective reference for the model; For the cross-dimensional interactive collaboration stage, a cross-dimensional information interaction mechanism is introduced; the 3x3 convolutional branch is responsible for extracting local multi-scale feature distributions, while the direction vector generated by the 1x1 branch is transformed into spatial weights through the Softmax layer, guiding the network to focus on important structural regions. The module compresses information from one branch using global average pooling and performs matrix multiplication with the original features of the other branch, thereby establishing a weight mapping between different feature scales; it also combines attention scores. Expressed as: , An organic link was established between local context and long-range spatial dependency; In the feature weighting and refinement output stage, the module fuses the attention features generated from the three branches and applies the sigmoid activation function. Generate the final multi-scale weight map ; Weights are applied to the grouped features through element-wise multiplication, and the enhanced feature output is obtained after dimensionality reduction. : 。 6. The method as described in claim 5, characterized in that, The visual distraction detection fused with a large language model is as follows: First, by transmitting real-time semantic segmentation results, the visual large language model is driven to deeply analyze the external scene and dynamically determine the important areas that the driver needs to focus on in the driving task; then, by combining the driver's gaze point detection results and verifying the consistency between the driver's gaze point area and the important areas, visual distraction determination is achieved.

7. A driver visual distraction detection system integrating external traffic environment perception, characterized in that: A driver visual distraction detection method for implementing the fusion of external traffic environment perception as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the steps in the driver visual distraction detection method that integrates external traffic environment perception as described in any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the driver visual distraction detection method that integrates external traffic environment perception as described in any one of claims 1 to 6.

Images