Driving intention recognition method and device based on visual attention and view field target change

Abstract

The invention discloses a driving intention recognition method and device based on visual attention and view field target change, relates to the field of driver intention perception and recognition, and predicts driver behaviors based on a deep learning calculation framework. A double-branch model architecture is provided, visual attention of a driver is considered, visual behavior characteristics under different operation intentions are described by establishing an eye movement mode pattern model, meanwhile, changes of a view field target of the driver are considered, and a time sequence characteristic calculation and extraction module is established by establishing a time section and time slice model, so that the visual behavior characteristics under different operation intentions are described. And a driving intention perception and identification calculation framework is formed. And building a deep learning environment training model in a PyTorch platform, and verifying the effectiveness of the training model on a test set. The technology provides reference and support for guaranteeing road traffic safety and developing an intelligent driving system.

Description

Technical Field

[0001] This invention relates to the field of driving intention perception and recognition, and more particularly to a method and apparatus for predicting driver behavior based on a dual-branch deep learning model framework, specifically a method and apparatus for driving intention recognition based on visual attention and changes in the field of view. Background Technology

[0002] The widespread adoption of automobiles has improved the efficiency and convenience of travel, but it has also increased the complexity of the traffic environment. The increasingly frequent interactions between vehicles pose a serious challenge to road safety. In all traffic accidents, over 80% can be attributed to driver error. The driver's state and behavior are crucial factors affecting driving safety. Therefore, learning and understanding driver intentions, predicting their actions, and providing warnings when necessary can reduce the probability of traffic accidents and maintain road safety.

[0003] While ensuring safety, the intelligentization of automobiles is also one of the main directions of future development. Existing Advanced Driving Assistance Systems (ADAS) typically operate independently of the driver's state or intentions, making decisions without considering the driver's expected actions. However, the driver is at the center of the "traffic-driver-vehicle" loop, and the driver's short-term decisions are the result of interactions with other road users or road infrastructure. Timely monitoring of driving intentions, combined with vehicle operating status and the surrounding traffic environment to predict potential hazards, not only helps to fully realize the effectiveness of ADAS but also reduces decision-making conflicts between ADAS and the driver.

[0004] Furthermore, inferring driving intentions and predicting driving behavior are crucial for the collaborative work between the driver and the autonomous driving system. Current autonomous driving technologies involve a handover process between the driver and the controller. Accurate inference of driving intentions allows for a smoother and safer transition between the driver and the system. Simultaneously, driving behavior is a complex process of making decisions and executing actions after a deep perception and cognitive understanding of the surrounding traffic environment. Improving the intelligence and reliability of autonomous driving requires the introduction of brain-like intelligence. Therefore, learning the process by which drivers generate operational intentions can provide effective support for building intelligent decision-making systems for autonomous driving.

[0005] Current research on driving intention recognition primarily focuses on the intentions of various short-term operational tasks, such as turning, changing lanes, accelerating, and braking. Most common driving intention recognition systems are based on three types of signal inputs: vehicle dynamic signals, traffic environment signals, and driver physiological signals. Among these, vehicle dynamic signals typically change significantly only after the driver's action and are easily affected by environmental changes, exhibiting a delay in information delivery, thus failing to provide timely information for intention prediction. From the perspective of driver cognition, intention recognition methods that integrate traffic environment information are currently a hot research topic. In the process of driver interaction with the environment, the visual channel is the primary means of information acquisition. Changes in visual attention precede driving behavior and can effectively characterize the driver's cognitive processing of the environment and anticipated vehicle actions. Therefore, by studying driver eye-tracking data, we can delve into their attention allocation patterns and cognitive processes in different driving scenarios, thereby improving the accuracy of driving intention recognition.

[0006] When recognizing driver intent, different methods are typically used to process traffic environment and driver visual signal data. Common methods include traditional machine learning and deep learning. Typical machine learning methods include discriminative and generative methods. Discriminative methods include Random Forest (RF) and Support Vector Machine (SVM), while generative methods include Hidden Markov Model (HMM). While these machine learning models can recognize driver intent, limitations in model size and training complexity mean that some models are weak at modeling long-term temporal dependencies and struggle with high-dimensional, large-scale data, resulting in poor generalization. In contrast, deep learning methods, with their complex network structures, possess powerful feature extraction capabilities. For example, Recurrent Neural Networks (RNNs) perform well in processing temporal data, leading to the development of many RNN-based neural network models in driver intent recognition. However, the interpretability and scalability of conventional deep learning need improvement, requiring further optimization by incorporating driver cognitive processes. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention provides a driving intention recognition method and device based on visual attention and field of view target changes, which realizes effective recognition and judgment of the driver's driving intention through a dual-branch network framework.

[0008] According to one aspect of the present invention, a method for recognizing driving intentions based on visual attention and changes in the field of view is provided, comprising: Based on the driver's eye movement data and driving scene data during driving, combined with the basic visual behavior of the human eye, a driving eye movement pattern graph model is constructed to extract the eye movement behavior features. At the same time, combined with the changes in the scene during driving, a segment-fragment time series model is established to extract the temporal features, thus forming complete driving eye movement data. The acquired driver eye-tracking data is input into a trained dual-branch parallel deep learning model, which outputs a driver intention classification and recognition result; wherein, the training of the multi-classification dual-branch parallel deep learning model includes: Construct a typical lateral driving intention driving eye-tracking sample dataset, and divide the sample dataset into a training set and a test set; A deep learning model with dual-branch parallel computing is constructed. The model captures the driver's visual attention features through graph attention calculation strategy and captures the driving field of vision target change features through long short-term memory mechanism. A decision-level fusion mechanism is adopted to integrate the classification calculation results of the two branches. The model was trained and validated using the training set and test set respectively, resulting in a trained dual-branch driving intention recognition deep learning model.

[0009] As a further technical solution, based on the driver's eye movement data and driving scenario data during driving, combined with basic human visual behavior, a driving eye movement pattern graph model is constructed, and eye movement behavior features are extracted, including: Using the point when the driver begins to perform driving operations as the boundary, eye movement data within the time window is extracted forward as segments of driving intentions. Based on the two visual behaviors of human eye fixation and saccade, the eye movement data in the driving intention segment are represented by nodes in the graph structure to represent fixation points and adjacent edges in the graph structure to represent saccade behaviors. Connect the 7th-order neighborhood nodes of each gaze point as a description of the local series of gaze behaviors; Each gaze point is spatially segmented, and a square area within 200 pixels around the gaze point is cropped as the visual attention focus area; The spatial coordinates of the gaze point are used as the visual main features of the graph node; Using a semantic segmentation model, the semantic category of the gaze point is matched as the visual target feature of the graph node; A convolutional neural network for image feature extraction is used to obtain image features of the visual attention focus area, which are then used as visual scene features for graph nodes. Calculate the Euclidean distance and vector direction angle between adjacent gaze points, and use them as the features of adjacent edges in the graph; Based on the extracted node features and adjacent edge features, a driver eye-tracking pattern graph model is obtained.

[0010] As a further technical solution, a segment-fragment time series model is established by combining the changes in the driving scene, and the temporal features are extracted to form complete driving eye-tracking data, including: The driving scene video in the driving intention segment is segmented into a series of image frames, and each image frame is the raw data of a time segment; Based on the foveal receptive field and peripheral receptive field mechanism in human visual perception, the complete data of each image frame is used as the global context, and a square area within 200 pixels around the gaze point on each image frame is cropped as the local scene. The extracted time slices are arranged in chronological order and filled and aligned to the same length to form a time segment model of the target in the driving field of view. Deep learning neural networks are used to perform convolution calculations on global context and local scene images, and the image features of both are extracted as temporal cross-section features. Deep learning neural networks are used to perform sequence calculations on time segments and extract their temporal features as time segment features. Based on the extracted time section features and time segment features, a section-segment time series model of target changes in the driving field of vision is obtained; The obtained driving eye-tracking pattern model is combined to form complete driving eye-tracking data.

[0011] As a further technical solution, the construction of the dual-branch parallel computing deep learning model includes: Establish a graph attention model branch for learning driver visual attention; Establish a long short-term memory model branch for learning temporal changes in the field of view target; A decision-level fusion mechanism is adopted, adaptive learning parameters are set, computational weights are allocated to the two branches, and the computational results of the two branches are fused.

[0012] As a further technical solution, training the dual-branch parallel computing deep learning model further includes: Set and adjust the model's hyperparameters, and train the model using the cross-entropy loss function, the Adam gradient descent optimization algorithm, and the learning rate scheduler; Analyze the model training effect based on the loss value change curves and recognition accuracy curves on the training and test sets to obtain the model that achieves the best recognition performance.

[0013] As a further technical solution, the method also includes: evaluating the model performance using confusion matrix, accuracy, precision, recall and F1 score to verify the effectiveness of the training results.

[0014] According to one aspect of the present invention, a driving intention recognition device based on visual attention and changes in field of view targets is provided, comprising: The preprocessing module is used to construct a driving eye movement pattern model based on the driver's eye movement data and driving scene data during driving, combined with the basic visual behavior of the human eye, and extract the eye movement behavior features. At the same time, it combines the changes in the scene during driving to establish a segment-fragment time series model, extract the temporal features, and form complete driving eye movement data. The intent classification and recognition module is used to input the acquired driving eye-tracking data into a trained dual-branch parallel computing deep learning model and output driving intent classification and recognition results; wherein, the training of the multi-classification dual-branch parallel computing deep learning model includes: Construct a typical lateral driving intention driving eye-tracking sample dataset, and divide the sample dataset into a training set and a test set; A deep learning model with dual-branch parallel computing is constructed. The model captures the driver's visual attention features through graph attention calculation strategy and captures the driving field of vision target change features through long short-term memory mechanism. A decision-level fusion mechanism is adopted to integrate the classification calculation results of the two branches. The model was trained and validated using the training set and test set respectively, resulting in a trained dual-branch driving intention recognition deep learning model.

[0015] As a further technical solution, the construction of the dual-branch parallel computing deep learning model includes: Establish a graph attention model branch for learning driver visual attention; Establish a long short-term memory model branch for learning temporal changes in the field of view target; A decision-level fusion mechanism is adopted, adaptive learning parameters are set, computational weights are allocated to the two branches, and the computational results of the two branches are fused.

[0016] According to one aspect of the present invention, an electronic device is provided, comprising: at least one processor, at least one memory, and a communication interface; wherein the processor, memory, and communication interface communicate with each other; the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to perform the method described thereon.

[0017] According to one aspect of the present invention, a non-transitory computer read storage medium is provided, the non-transitory computer read storage medium storing computer instructions that cause the computer to perform the method described herein.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention establishes a data model based on driver eye-tracking data and driving scene videos. It uses gaze-saccade connection edges to construct an eye-tracking pattern graph model and segments to construct a time-series model, describing data related to driving intentions. Simultaneously, a two-branch deep learning neural network structure is built, using the driver eye-tracking pattern graph model and the driving field-of-view target time-series model as inputs. By capturing the driver's visual attention allocation patterns and the changing characteristics of driving field-of-view targets, it achieves the perception and recognition of driving intentions such as lane changes and turns, providing support for ensuring road traffic safety and developing intelligent driving systems. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a driving intention recognition method based on visual attention and changes in the field of view, provided in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the training process of a dual-branch parallel computing deep learning model provided in an embodiment of the present invention.

[0022] Figure 3 The process of modeling and feature extraction of driving eye movement patterns provided in the embodiments of the present invention.

[0023] Figure 4 The driving field of view target time series modeling and feature extraction process provided in the embodiments of the present invention.

[0024] Figure 5 This is a schematic diagram of the dual-branch parallel computing deep learning model structure provided in an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram of the confusion matrix results of the test set provided in an embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram of a driving intention recognition device based on visual attention and changes in the field of view, provided as an embodiment of the present invention.

[0027] Figure 8 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0028] During the formation of driving intention, drivers capture visual information and adjust their attention distribution through eye movements. Stable fixation and rapid saccades together form the basis of visual attention. Therefore, modeling these two visual behaviors as a graph structure composed of nodes and adjacent edges effectively preserves complete and accurate visual attention information. Simultaneously, the vehicle's environment, as input to the driver's visual system, exhibits dynamic changes during driving. This dynamism is reflected in the changing driving environment with the vehicle's position, and the dynamic adjustment of the driver's focus target based on driving needs. Therefore, temporal modeling of the driver's field of view and the fixation target data during the intention stage effectively learns and captures dynamically changing features and behavioral patterns, and uses historical information to predict future states. Combining these two approaches, the eye-tracking pattern graph model of visual attention and the temporal model of the field of view target are used as inputs to a neural network. These are processed and computed using two divide-and-conquer methods, respectively. This not only analyzes the driver's cognitive decision-making process before performing specific operations but also provides strong support for maintaining road traffic safety and improving vehicle driving intelligence.

[0029] Based on the above, this invention provides a driving intention recognition method based on visual attention and changes in the field of view target. By constructing an eye-tracking pattern graph model, the driver's eye-tracking features are extracted to describe visual attention. By constructing a time series model, global and local information in time sections and time segments are extracted to describe the changes in the field of view and target. Based on this, a dual-branch network that can learn the driver's visual patterns and the changing patterns of the driving environment is constructed to achieve effective perception and recognition of driving intention.

[0030] This invention utilizes deep learning methods to solve the problem of driver intention perception and recognition. Therefore, the following content will mainly introduce the relevant theoretical knowledge of deep learning used in this invention, including Graph Attention Network (GAT) and Long Short-Term Memory (LSTM).

[0031] Graph Attention Neural Networks (PANNs) introduce an attention mechanism based on graph convolution computation. They achieve weighted aggregation of neighbor node features through dynamically generated attention coefficients, thereby learning new feature representations for the central node. This network uses a point-by-point computation mode instead of traditional Laplacian matrix operations, overcoming the limitations of fixed operators and effectively handling complex topologies such as directed graphs. Simultaneously, a multi-head attention mechanism is implemented to learn multiple sets of attention weights in parallel, capturing local and global relationships between nodes in different representation subspaces, significantly improving the feature extraction capability for graph data.

[0032] Long Short-Term Memory (LSTM) networks are structural improvements on recurrent neural networks (RNNs). They utilize gating units to control the transmission and forgetting of information, effectively addressing the limitations of RNNs in handling long sequences. LSM networks employ three key gating units: the input gate, the forget gate, and the output gate. The input gate determines which information from the current time step is input to the cell state; the forget gate determines which information from the previous time step is forgotten by the cell state; and the output gate determines which information from the current time step's cell state is passed to the hidden state of the next time step or used as the model's output. Through this gating mechanism, LSM networks selectively retain and forget historical information, significantly enhancing their ability to process time-series data and effectively resolving the vanishing and exploding gradient problems.

[0033] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0036] This invention provides a method for recognizing driving intentions based on visual attention and changes in the field of view, such as... Figure 1 As shown, the process includes: constructing a driving eye movement pattern model based on the driver's eye movement data and driving scene data during driving, combined with basic human visual behavior, extracting eye movement behavior features, and simultaneously establishing a time series model based on changes in the driving scene, extracting temporal features to form complete driving eye movement data; inputting the obtained driving eye movement data into a trained dual-branch parallel deep learning model, and outputting driving intention classification and recognition results.

[0037] The training of the multi-class dual-branch parallel deep learning model, such as... Figure 2 As shown, it includes: Step 1: Construct a typical lateral driving intention driving eye-tracking sample dataset, and divide the sample dataset into a training set and a test set; Step 2: Build a deep learning model with dual-branch parallel computing. Capture the driver's visual attention features through graph attention computing strategy and capture the driving field of vision target change features through long short-term memory mechanism. Then, adopt a decision layer fusion mechanism to integrate the classification computing results of the two branches. Step 3: Train and validate the model using the training set and test set respectively to obtain the trained dual-branch driving intention recognition deep learning model.

[0038] In the embodiments of the present invention, when modeling and extracting features from driving data, an eye-tracking pattern graph data model is established using graph nodes and adjacent edges based on human eye gaze and saccade behavior, and visual attention features are extracted from it; by defining the time sections and time segments of the global context and local scenes in driving, and extracting the temporal features of changes in the field of view targets, the data source of the model is formed.

[0039] As a preferred embodiment, the driving eye-tracking pattern modeling and feature extraction of this invention, such as... Figure 3 As shown, the specific steps are as follows: Step 1.1: Using the time point when the driver begins to perform driving operations as the boundary, extract eye movement data within the time window in advance as driving intention segments.

[0040] Step 1.2: Based on the two visual behaviors of human eye, fixation and saccade, the eye movement data in the driving intention segment are represented by nodes in a graph structure to indicate fixation points, and adjacent edges in the graph structure to indicate saccade behaviors.

[0041] Step 1.3: Connect the 7th-order neighborhood nodes of each gaze point as a description of the local series of gaze behaviors.

[0042] Step 1.4: Spatial segmentation is performed on each gaze point, and a square area within 200 pixels around the gaze point is cropped as the visual attention focus area.

[0043] Step 1.5: After standardizing the spatial coordinates of the gaze point, it is used as the visual subject feature of the graph node.

[0044] Step 1.6: Using the Deeplab v3+ semantic segmentation model, the semantic category is matched by the coordinates of the gaze point, and the encoded result is used as the visual target feature of the graph node.

[0045] Step 1.7: Use the ResNet50 deep learning neural network to perform convolution calculations on the visual attention focal area to obtain image features, which are then used as visual scene features for graph nodes.

[0046] Step 1.8: Calculate the Euclidean distance and vector direction angle tanh value between adjacent gaze points, and use them as adjacent edge features in the graph.

[0047] Step 1.9: Based on the extracted node features and adjacent edge features, a driver eye-tracking pattern graph model is obtained.

[0048] As a preferred embodiment, the driving field of view target time series modeling and feature extraction of this invention, such as... Figure 4 As shown, the specific steps are as follows: Step 2.1: Segment the driving scene video in the driving intention segment into a series of image frames, with each image frame serving as raw data for a time segment.

[0049] Step 2.2: Based on the foveal receptive field and peripheral receptive field mechanism in human visual perception, the complete data of each image frame is used as the global context, and a square area within 200 pixels around the gaze point on each image frame is cropped as the local scene.

[0050] Step 2.3: Arrange the extracted time slices in chronological order and fill and align them to the same length to form a time segment model of the target in the driving field of view.

[0051] Step 2.4: Use the ResNet50 deep learning neural network to perform convolution calculations on the global context and local scene images, and extract their image features as temporal cross-section features.

[0052] Step 2.5: Use the LSTM deep learning neural network to perform sequence calculations on the time segments and extract their temporal features as time segment features.

[0053] Step 2.6: Based on the extracted time section features and time segment features, obtain the section-segment time series model of target changes in the driving field of vision.

[0054] Step 2.7: Combine the driving eye movement pattern model obtained in Step 1.9 to form complete driving eye movement data.

[0055] As a preferred embodiment, the present invention provides a dual-branch parallel computing deep learning model and environment, such as... Figure 5 As shown, this includes establishing graph attention model branches and long short-term memory model branches, as well as adding a decision-level fusion mechanism. The specific steps are as follows: Step 3.1: Establish a graph attention calculation model, which includes three GATConv layers. By adding an attention mechanism to the graph convolution calculation, the model can allocate attention weights according to the importance of different node features, highlighting eye movement patterns that are strongly related to driving intentions.

[0056] Step 3.2: Establish a long short-term memory (LSTM) computation model, which includes two LSTM layers. By using a gating mechanism at each time step to calculate and transmit information, the model can effectively memorize and integrate the temporal evolution of targets in the driving field of vision.

[0057] Step 3.3: Add a decision-level fusion mechanism, set adaptive learning parameters, and dynamically adjust the weights of the graph attention calculation model branch and the long short-term memory model branch to achieve optimal fusion of eye-tracking pattern features and field-of-view target temporal features.

[0058] Step 3.4: Customize the required environment using the PyTorch deep learning experimental environment library.

[0059] As a preferred embodiment, the model training and training result analysis of this invention employs the cross-entropy loss function, the Adam gradient descent optimization algorithm, and a learning rate scheduler to train and debug the training set of experimental data. The training results are then analyzed on the test set to confirm the effectiveness of the model training and complete the identification and judgment of the driver's driving intention. The specific steps are as follows: Step 4.1: Import the processed typical driving intention eye-tracking sample dataset to prepare the training set and test set, with a ratio of approximately 8:2.

[0060] Step 4.2: Import the model built in Step 3, set the model's hyperparameters, including learning rate, learning rate decay rate, number of neurons in the hidden layer, etc., and use the cross-entropy loss function, Adam gradient descent optimization algorithm, and learning rate scheduler to train the model. Step 4.3: Analyze the model training effect based on the loss value change curve and the recognition accuracy change curve to obtain the model with the best recognition performance. It should be noted that the model's optimal recognition performance is evaluated by combining the loss value change curve and the recognition accuracy change curves on the training and test sets. When the test set loss decreases to a certain level and then begins to rise, or the training set accuracy increases while the test set accuracy decreases or remains essentially unchanged, the model has already exhibited overfitting. The model with the highest test set accuracy is then defined as the optimal driving intention recognition model.

[0061] Step 4.4, as follows Figure 6 As shown, the confusion matrix, accuracy, precision, recall, and F1 score are used to evaluate the model performance and verify the effectiveness of the training results.

[0062] The implementation of the various embodiments of the present invention is based on programmed processing by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a driving intention recognition device based on visual attention and changes in the field of view target. This device is used to execute a driving intention recognition method based on visual attention and changes in the field of view target in the above method embodiments.

[0063] See Figure 7 The device includes: The preprocessing module is used to construct a driving eye movement pattern model based on the driver's eye movement data and driving scene data during driving, combined with the basic visual behavior of the human eye, and extract the eye movement behavior features. At the same time, it combines the changes in the scene during driving to establish a time series model and extract the temporal features to form complete driving eye movement data. The intent classification and recognition module is used to input the obtained driving eye-tracking data into a trained dual-branch parallel computing deep learning model and output the driving intent classification and recognition result. The training of the multi-classification dual-branch parallel computing deep learning model includes: constructing a driving eye-tracking sample dataset of typical lateral driving intents and dividing the dataset into a training set and a test set; building a dual-branch parallel computing deep learning model, capturing driver visual attention features through graph attention calculation strategies and capturing driving field-of-view target change features through long short-term memory mechanisms, and employing a decision-level fusion mechanism to integrate the classification calculation results of the two branches; and training and validating the model using the training set and test set respectively to obtain a trained dual-branch driving intent recognition deep learning model.

[0064] The driving intention recognition device based on visual attention and field-of-view target changes provided in this invention uses a two-branch deep learning framework to understand and judge the driver's driving intention. Figure 7 Several modules are used to establish eye-tracking pattern models and temporal change models of visual targets under different driving intentions, extracting visual attention features and temporal features of visual target changes. A two-branch deep learning environment is built in the PyTorch deep learning experimental library to train the model, and the effectiveness of the built model is verified on the test set. This device can be applied to the perception and recognition of driver intentions, providing reference and support for advanced driver assistance systems (ADAS) and autonomous driving systems in the decision-making and control process.

[0065] It should be noted that the device embodiments provided by the present invention, in addition to implementing the methods in the above method embodiments, are also used to implement the methods in other method embodiments provided by the present invention. The difference lies only in the setting of corresponding functional modules. Their principles are basically the same as those of the above device embodiments provided by the present invention. Anyone skilled in the art, based on the above device embodiments and referring to the specific technical solutions in other method embodiments, can obtain corresponding technical means and technical solutions constituted by combining technical features, and, while ensuring the practicality of the technical solutions, can improve the device in the above device embodiments to obtain corresponding device-type embodiments for implementing the methods in other method-type embodiments. For example: Based on the above-described device embodiments, as a preferred embodiment, the driving intention recognition device based on visual attention and field-of-view target changes provided in this invention further includes: The first processing submodule is used to extract eye movement data within a time window, starting from the point when the driver begins to perform driving operations, as a segment of driving intention. The second processing submodule is used to represent fixation points and saccadic behavior in eye movement data in driving intention segments using nodes in a graph structure and adjacent edges in a graph structure, based on the two visual behaviors of human eye: fixation and saccadic behavior. The third processing submodule is used to connect the 7th-order neighborhood nodes of each gaze point as a description of the local series of gaze behaviors; The fourth processing submodule is used to spatially segment each gaze point and crop a square area within 200 pixels around the gaze point as the visual attention focus area. The fifth processing submodule is used to use the spatial coordinates of the gaze point as the visual subject feature of the graph node; The sixth processing submodule is used to match the semantic category of the gaze point using a semantic segmentation model, which is then used as the visual target feature of the graph node. The seventh processing submodule is used to obtain image features of the visual attention focus area using a convolutional neural network for image feature extraction, and use them as visual scene features of graph nodes. The eighth processing submodule is used to calculate the Euclidean distance and vector direction angle between adjacent gaze points, as the adjacent edge features in the graph; Based on the above-described device embodiments, as a preferred embodiment, the driving intention recognition device based on visual attention and field-of-view target changes provided in this invention further includes: The ninth processing submodule is used to segment the driving scene video in the driving intent segment into a series of image frames, with each image frame serving as raw data for a time segment; The tenth processing submodule is used to take the complete data of each image frame as the global context based on the foveal receptive field and peripheral receptive field mechanism in human visual perception, and to crop the square area within 200 pixels around the gaze point on each image frame as the local scene. The eleventh processing submodule is used to arrange the extracted time slices in chronological order and fill and align them to the same length to form a time segment model of the target in the driving field of vision. The twelfth processing submodule is used to perform convolution calculations on global context and local scene images using deep learning neural networks, and extract their image features as temporal cross-section features. The thirteenth processing submodule is used to perform sequence calculations on time segments using deep learning neural networks and extract their temporal features as time segment features. Based on the above-described device embodiments, as a preferred embodiment, the driving intention recognition device based on visual attention and field-of-view target changes provided in this invention further includes: The first model building submodule is used to build a graph attention model branch for learning driver visual attention; The second model building submodule is used to establish a long short-term memory model branch for learning the temporal changes of the target in the field of view; The third model building submodule is used to adopt a decision-level fusion mechanism, set adaptive learning parameters, allocate computational weights for the two branches, and fuse the computational results of the two branches.

[0066] Based on the above-described device embodiments, as a preferred embodiment, the driving intention recognition device based on visual attention and field-of-view target changes provided in this embodiment of the invention further includes: The first training submodule is used to set and adjust the hyperparameters of the model, and to train the model using the cross-entropy loss function, the Adam gradient descent optimization algorithm, and the learning rate scheduler. The second training submodule is used to analyze the model training effect based on the loss value change curve and recognition accuracy curve on the training set and test set, and to obtain the model that achieves the best recognition performance.

[0067] Based on the above-described device embodiments, as a preferred embodiment, the driving intention recognition device based on eye-tracking modeling provided in this invention further includes: The third training submodule is used to evaluate the model performance using the confusion matrix, accuracy, precision, recall, and F1 score, and to verify the effectiveness of the training results.

[0068] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 8 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0069] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0070] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0071] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] Based on the same inventive concept as any of the foregoing embodiments, this embodiment of the invention also provides a non-transitory computer read storage medium, which stores computer instructions that cause the computer to execute the method described above.

[0076] In summary, this invention establishes an eye-tracking pattern model and a temporal change model of the target in the field of view based on visual behavior under different driving intentions, extracting visual attention features and temporal features of target changes in the field of view. A dual-branch deep learning environment was built in the PyTorch deep learning experimental library to train the model, and the effectiveness of the model was verified on a test set. This device can be applied to the perception and recognition of driver intentions, providing reference and support for advanced driver assistance systems (ADAS) and autonomous driving systems in the decision-making and control process.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A driving intention recognition method based on visual attention and field of view target change, characterized in that, include: Based on the driver's eye movement data and driving scene data during driving, combined with the basic visual behavior of the human eye, a driving eye movement pattern graph model is constructed to extract the eye movement behavior features. At the same time, combined with the changes in the scene during driving, a segment-fragment time series model is established to extract the temporal features, thus forming complete driving eye movement data. The acquired driver eye-tracking data is input into a trained dual-branch parallel deep learning model, which outputs a driver intention classification and recognition result; wherein, the training of the dual-branch parallel deep learning model includes: Construct a typical lateral driving intention driving eye-tracking sample dataset, and divide the sample dataset into a training set and a test set; A dual-branch parallel deep learning model was built, which captures the driver's visual attention features through graph attention calculation strategy and captures the driving field of vision target change features through long short-term memory mechanism. A decision layer fusion mechanism was adopted to integrate the classification calculation results of the two branches. The model is trained and validated using the training set and test set respectively, resulting in a trained dual-branch parallel deep learning model for driving intention recognition. 2.The driving intention recognition method based on visual attention and field of view target change according to claim 1, wherein, Based on driver eye movement data and driving scenario data during driving, and combined with basic human visual behavior, a driving eye movement pattern graph model is constructed, and eye movement behavior features are extracted, including: Using the point when the driver begins to perform driving operations as the boundary, eye movement data within the time window is extracted forward as segments of driving intentions. Based on the two visual behaviors of human eye fixation and saccade, the eye movement data in the driving intention segment are represented by nodes in the graph structure to represent fixation points and adjacent edges in the graph structure to represent saccade behaviors. Connect the 7th-order neighborhood nodes of each gaze point as a description of the local series of gaze behaviors; Each gaze point is spatially segmented, and a square area within 200 pixels around the gaze point is cropped as the visual attention focus area; The spatial coordinates of the gaze point are used as the visual main features of the graph node; Using a semantic segmentation model, the semantic category of the gaze point is matched as the visual target feature of the graph node; A convolutional neural network for image feature extraction is used to obtain image features of the visual attention focus area, which are then used as visual scene features for graph nodes. Calculate the Euclidean distance and vector direction angle between adjacent gaze points, and use them as the features of adjacent edges in the graph; Based on the extracted node features and adjacent edge features, a driver eye-tracking pattern graph model is obtained. 3.The driving intention recognition method based on visual attention and field of view target change according to claim 2, characterized in that, By combining the changes in the driving scene, a segment-fragment time series model is established, and the temporal features are extracted to form complete driving eye-tracking data, including: The driving scene video in the driving intention segment is segmented into a series of image frames, and each image frame is the raw data of a time segment; Based on the foveal receptive field and peripheral receptive field mechanism in human visual perception, the complete data of each image frame is used as the global context, and a square area within 200 pixels around the gaze point on each image frame is cropped as the local scene. The extracted time slices are arranged in chronological order and filled and aligned to the same length to form a time segment model of the target in the driving field of view. The global context and local scene image are calculated by using a deep learning neural network, and the image features of the two are extracted as time slice features; The time slice is calculated by using a deep learning neural network, and the time sequence features are extracted as time slice features; Based on the extracted time slice features and time slice features, a slice-segment time sequence model of the target change of the driving field of view is obtained; The driving eye movement data is formed by combining the driving eye movement pattern graph model.

4. The driving intention recognition method based on visual attention and field of view target change according to claim 1, characterized in that, The construction of the double-branch parallel computing deep learning model includes: A graph attention model branch is established for learning the visual attention of the driver; A long short-term memory model branch is established for learning the time sequence change of the field of view target; An adaptive learning parameter is set by using a decision layer fusion mechanism, and the calculation weights of the two branches are allocated to fuse the calculation results of the two branches.

5. The driving intention recognition method based on visual attention and field of view target change according to claim 4, characterized in that, When training the double-branch parallel computing deep learning model, it further includes: The hyperparameters of the model are set and adjusted, and the model is trained by using a cross-entropy loss function, an Adam gradient descent optimization algorithm and a learning rate scheduler; The model training effect is analyzed according to the loss value change curve and the recognition accuracy curve on the training set and the test set, and the model reaching the best recognition performance is obtained.

6. The driving intention recognition method based on visual attention and field of view target change according to claim 5, characterized in that, The method further includes: using a confusion matrix, accuracy, precision, recall and F1 score to evaluate the model performance and verify the effectiveness of the training result.

7. A driving intention recognition device based on visual attention and field of view target change, characterized by, It includes: A preprocessing module is used to construct a driving eye movement pattern graph model according to the eye movement data and driving scene data of the driver during driving, combine the basic visual behavior of the human eye, extract the eye movement behavior features, and at the same time, combine the changes of the scene during driving, establish a slice-segment time sequence model, extract the time sequence features, and form complete driving eye movement data; An intention classification and recognition module is used to input the obtained driving eye movement data into a trained double-branch parallel computing deep learning model, and output a driving intention classification and recognition result; wherein the training of the double-branch parallel computing deep learning model includes: A driving eye movement sample data set of typical lateral driving intention is constructed, and the sample data set is divided into a training set and a test set; A double-branch parallel computing deep learning model is constructed, which captures the visual attention features of the driver by using a graph attention calculation strategy, captures the driving field of view target change features by using a long short-term memory mechanism, and comprehensively integrates the classification calculation results of the two branches by using a decision layer fusion mechanism; The training set and the test set are used for model training and verification respectively, and a trained double-branch driving intention recognition deep learning model is obtained. 8.The driving intention recognition device based on visual attention and changes in field of view targets according to claim 7, characterized in that, The construction of the double-branch parallel computing deep learning model includes: A graph attention model branch is established for learning the visual attention of the driver; A long short-term memory model branch is established for learning the time sequence change of the field of view target; An adaptive learning parameter is set by using a decision layer fusion mechanism, and the calculation weights of the two branches are allocated to fuse the calculation results of the two branches.

9. An electronic device, comprising: It includes: At least one processor, at least one memory and a communication interface; wherein the processor, the memory and the communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor invokes the program instructions to execute the method of any one of claims 1 to 6.

10. A non-transitory computer readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method of any one of claims 1 to 6.

Images