Driver target attention prediction method, system, device, medium and product

Abstract

The invention discloses a driver target attention method, system and device, a medium and a product, and relates to the field of target detection, and the method comprises the steps: constructing a driver target attention prediction model; the driver target attention prediction model comprises a target recognition model, a driving event classification model and an attention prediction model; the attention prediction model comprises an environment model fusing multi-layer visual memory and a target-level visual attention prediction model; obtaining visual behavior data and driving data of a driver in real time; and inputting the driver visual behavior data and the driving data into the driver target attention prediction model to obtain a visual attention prediction target. According to the method, the adaptive capacity of the driver visual target attention prediction model can be improved, the accuracy and robustness of model prediction are improved, the prediction result better fits the visual search mechanism of the driver, and the practicability of the driver visual target attention prediction model is improved.

Description

Technical Field

[0001] The present application relates to the field of target detection, and in particular to a method, system, device, medium and product for driver target attention. Background Art

[0002] One of the primary ways drivers perceive their surroundings is through their visual perception of the environment. Driven by the demands of complex driving tasks, drivers dynamically allocate, actively shift, and select their visual attention, enabling them to filter high-value targets and information relevant to their current task in complex and changing environments (such as urban traffic and complex off-road environments) and make decisions based on this selected information. By analyzing, learning, and predicting the dynamic shift of a driver's visual attention to objects of interest, it is possible to determine the driver's current level and state of awareness of the surrounding environment (e.g., whether they observe high-risk objects in the environment, whether they are distracted by the driver), which is crucial for designing safe and reliable driver assistance systems.

[0003] Currently, most existing methods for predicting driver visual attention primarily use color and texture features, driving task requirements, or a combination of both as input to the prediction model. They output pixel-level probability heat maps, representing the probability distribution of the driver's visual attention across the entire visual area. However, pixel-based prediction methods only capture the static spatial distribution of the driver's attention, ignoring the dynamic nature of the driver's attention allocation process, making them difficult to apply to dynamic and complex scenarios.

[0004] Furthermore, in dynamic and complex scenes, traditional methods mostly use pixels as prediction output, and the output often covers multiple targets, resulting in unclear prediction results for the target of interest. This makes it difficult to align with the driver's visual attention allocation mechanism, resulting in low practicality. Therefore, in order to improve the adaptability of the driver's visual target attention prediction model in complex and dynamic scenes, enhance the accuracy and robustness of the model prediction, make the prediction results more consistent with the driver's visual search mechanism, and improve practicality, a driver target attention prediction method is urgently needed. Summary of the Invention

[0005] The purpose of this application is to provide a driver target attention prediction method, system, equipment, medium and product, which can improve the adaptability of the driver's visual target attention prediction model, improve the accuracy and robustness of the model prediction, make the prediction results more in line with the driver's visual search mechanism, and improve the practicality of the driver's visual target attention prediction model.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for predicting a driver's target attention, comprising:

[0008] Constructing a driver target attention prediction model; the driver target attention prediction model includes: a target recognition model, a driving event classification model, and an attention prediction model; the attention prediction model includes an environmental model integrated with multi-layer visual memory and a target-level visual attention prediction model; the target recognition model is used to obtain a target recognition result based on the driver's visual behavior data and driving data; the driving event classification model is used to obtain a driving event classification result based on the target recognition result and the driving data corresponding to the target recognition result; the environmental model integrated with multi-layer visual memory is used to obtain a coding feature vector based on the target recognition result and the driving event classification result; the target-level visual attention prediction model is used to obtain a visual attention prediction target based on the coding feature vector;

[0009] Acquire driver's visual behavior data and driving data in real time;

[0010] The driver's visual behavior data and the driving data are input into the driver's target attention prediction model to obtain the visual attention prediction target.

[0011] Optionally, the constructing of the driver's target attention prediction model includes:

[0012] Constructing a first sample data set and the target recognition model; the first sample data set includes driver visual behavior history data, driving history data, and annotated data corresponding to the driver visual behavior history data and the driving history data; the driver visual behavior history data includes: first-person perspective video, binocular iris positioning, and viewpoint projection coordinates; the driving history data includes: lidar point cloud and vehicle-mounted camera images;

[0013] Using the first sample data set to train and test the target recognition model until a first set condition is met, and using the trained target recognition model as a trained target recognition model;

[0014] Constructing a second sample data set and the driving event classification model; the second sample data set includes annotated data corresponding to the driver's visual behavior history data and the driving history data;

[0015] using the second sample data set to train and test the driving event classification model until a second set condition is met, and using the trained driving event classification model as a trained driving event classification model;

[0016] Constructing a third sample data set and the attention prediction model; the third sample data set includes annotated data corresponding to the driver's visual behavior history data and the driving history data; the attention prediction model includes the environment model fused with multi-layer visual memory and the target-level visual attention prediction model;

[0017] The driving event classification model is trained and tested using the third sample data set until a third set condition is met, and the trained attention prediction model is used as the trained attention prediction model.

[0018] Optionally, constructing the first sample data set includes:

[0019] Obtaining, based on the driver's visual behavior history data and the driving history data, labeled data corresponding to the driver's visual behavior history data and the driving history data to generate a training sample pair;

[0020] A first sample data set is constructed based on the training sample pairs.

[0021] Optionally, constructing the driving event classification model includes: adopting a nearest neighbor-based hierarchical clustering method to obtain a classification result of the driving event according to the labeled data corresponding to the driver's visual behavior history data and the driving history data.

[0022] Optionally, the fused multi-layer visual memory environment model includes: a current scene layer, a history sequence layer, and an attention trajectory layer;

[0023] Obtaining the current scene layer based on the target recognition model and the driving event classification model; obtaining the historical sequence layer based on the current scene layer;

[0024] The attention trajectory layer is used to record the results of the current scene layer based on the historical sequence layer.

[0025] Optionally, the target-level visual attention prediction model includes: a multi-layer perceptron;

[0026] The multi-layer perceptron is used to decode the encoded feature vector and generate the visual attention prediction target.

[0027] In a second aspect, the present application provides a driver target attention prediction system, comprising:

[0028] Driving data collection module, used to collect driver visual behavior data and driving data in real time;

[0029] a target recognition module, configured to process the driver's visual behavior data and the driving data to obtain a target recognition result;

[0030] a driving event classification module, configured to process the driving data based on the target recognition result to obtain a driving event classification result;

[0031] The attention prediction module is used to obtain a visual attention prediction target based on the target recognition result and the driving event classification result.

[0032] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned methods for predicting driver target attention.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for predicting driver target attention.

[0034] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for predicting driver target attention.

[0035] According to the specific embodiments provided in this application, this application has the following technical effects:

[0036] The present application provides a method, apparatus, device, medium and product for predicting driver's target attention. By constructing a target recognition model and a driving event classification model, the driver's target attention prediction model can better match the driver's visual attention allocation mechanism, thereby improving the practicality of the driver's target attention prediction method. By constructing an attention prediction model, the attention prediction model includes an environmental model integrating multi-layer visual memory and a target-level visual attention prediction model, thereby improving the adaptability of the driver's target attention prediction model, making the prediction results more in line with the driver's visual search mechanism, and being able to improve the accuracy of the prediction results in multi-target scenarios, thereby improving the accuracy and robustness of the model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 This is a flowchart of a method for predicting driver's target attention in one embodiment of the present application;

[0039] Figure 2 A structural diagram of a driver's target attention prediction system provided in one embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0041] Figure numerals: 1-driving data acquisition module, 2-target recognition module, 3-driving event classification module, 4-attention prediction module. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The purpose of this application is to provide a driver target attention prediction method, system, equipment, medium and product, which can improve the adaptability of the driver's visual target attention prediction model, enhance the accuracy and robustness of the model prediction, make the prediction results more in line with the driver's visual search mechanism, and improve practicality.

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0045] In an exemplary embodiment, a method for predicting a driver's target attention is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a computer device such as a terminal and a server. Figure 1 As shown, the method includes:

[0046] Step 10: Construct a driver target attention prediction model. The driver target attention prediction model includes a target recognition model, a driving event classification model, and an attention prediction model. The attention prediction model includes an environment model integrated with multi-layer visual memory and a target-level visual attention prediction model. The target recognition model is used to obtain target recognition results based on the driver's visual behavior data and driving data. The driving event classification model is used to obtain driving event classification results based on the target recognition results and the driving data corresponding to the target recognition results. The environment model integrated with multi-layer visual memory is used to obtain an encoded feature vector based on the target recognition results and the driving event classification results. The target-level visual attention prediction model is used to obtain a visual attention prediction target based on the encoded feature vector.

[0047] Step 10 specifically includes:

[0048] Step 11: Obtain the driver's visual behavior history data and driving history data. The driver's visual behavior history data includes first-person video, binocular iris positioning, and viewpoint projection coordinates. The driving history data includes lidar point clouds and vehicle camera images. Based on the driver's visual behavior history data and driving history data, labeled data corresponding to the driver's visual behavior history data and driving history data is obtained to generate training sample pairs.

[0049] Step 12: Construct a first sample data set and a target recognition model. The first sample data set is constructed based on the training sample pairs. The first sample data set includes the driver's visual behavior history data, driving history data, and annotated data corresponding to the driver's visual behavior history data and driving history data.

[0050] Step 13: Use the first sample data set to train and test the target recognition model until the first set condition is met, and use the trained target recognition model as the trained target recognition model.

[0051] Step 14: Construct a second sample data set and a driving event classification model. The second sample data set is constructed based on the training sample pairs. The second sample data set includes annotated data corresponding to the driver's visual behavior history data and driving history data.

[0052] Step 15: Use the second sample data set to train and test the driving event classification model until a second set condition is met, and use the trained driving event classification model as the trained driving event classification model.

[0053] Step 16: Construct a third sample dataset and an attention prediction model. The third sample dataset is constructed based on the training sample pairs. The third sample dataset includes labeled data corresponding to the driver's visual behavior history data and driving history data. The attention prediction model includes an environment model that integrates multiple layers of visual memory and an object-level visual attention prediction model.

[0054] Step 17: Use the third sample data set to train and test the attention prediction model until the third set condition is met, and use the trained attention prediction model as the trained attention prediction model.

[0055] Step 20: Acquire the driver's visual behavior data and driving data in real time.

[0056] Step 30: Input the driver's visual behavior data and driving data into the driver's target attention prediction model to obtain a visual attention prediction target.

[0057] For example, the obtained driver visual behavior data and driving data are input into the driver target attention prediction model, specifically: the driver visual behavior data and driving data are input into the target recognition model to obtain the target recognition result; the target recognition result and the driving data corresponding to the target recognition result are input into the driving event classification model to obtain the driving event classification result; the target recognition result and the driving event classification result are input into the environment model integrating multi-layer visual memory to obtain the encoding feature vector; the encoding feature vector is input into the target-level visual attention prediction model to obtain the visual attention prediction target.

[0058] Implementing the above steps can improve the adaptability of the driver's visual target attention prediction model, enhance the accuracy and robustness of the model's prediction, make the prediction results more consistent with the driver's visual search mechanism, and improve the practicality of the driver's visual target attention prediction model.

[0059] As an optional implementation method, in order to improve the practicality of the driver's target attention prediction model and enable it to better match the driver's visual attention allocation mechanism, obtaining the driver's visual behavior history data and driving history data specifically includes: obtaining the driver's visual behavior history data and driving history data under various driving scenarios (including driver control history data, vehicle status history data, and driving environment history data); selecting the driving history data of multiple drivers under different driving scenarios, including straight roads, intersections, T-junctions, roundabouts, and ramps.

[0060] The driver's visual behavior history data obtained includes: first-person video, binocular iris positioning, and viewpoint projection coordinates. The binocular iris positioning results are used to calculate the coordinate position of the driver's visual focus in the first-person video. The driving history data obtained includes on-board sensor data (including lidar point cloud, on-board camera images, satellite positioning, vehicle heading, and vehicle acceleration) and driving operation history data (including steering wheel angle, accelerator and brake pedal travel, and gear position).

[0061] The labeled data corresponding to the driver's visual behavior history data and driving history data are obtained based on the driver's visual behavior history data and driving history data, specifically including:

[0062] (1) Align and correct the first-person perspective video in the driver's visual behavior history data with the on-board camera image in the driving history data, and project the on-board camera image into the spatial coordinate system of the lidar point cloud in the driving history data.

[0063] (2) Mark the target to be identified on the vehicle-mounted camera image. The target categories of the target to be identified include pedestrians, non-motor vehicles, motor vehicles, road signs and landmark lines.

[0064] (3) The coordinate position of the driver's visual focus in the first-person video is projected onto the vehicle-mounted camera image, and the driver's visual focus (including focused and ignored) is added to the target attributes of the target to be identified. The target focused by the driver's visual focus is taken as the true value result.

[0065] (4) The estimated results of the target’s relative position, motion speed, and heading in the spatial coordinate system of the lidar point cloud are added to the target attributes of the target to be identified, which are used to describe the spatial motion characteristics of the target to be identified and identify driving events between two targets.

[0066] As an optional implementation, in order to improve the accuracy of the target recognition model, YOLOV5 is used as the target recognition algorithm of the target recognition model. The target recognition model is trained and tested using the first sample data set until the first set condition is met, and the trained target recognition model is used as the trained target recognition model.

[0067] As an optional implementation, in order to improve the accuracy of the driving event classification model, a hierarchical clustering method based on nearest neighbor is adopted to obtain the classification results of driving events based on the labeled data corresponding to the driver's visual behavior history data and the driving history data.

[0068] For example, based on the annotated data corresponding to the driver's visual behavior history data and the driving history data, the target categories (including pedestrians, non-motor vehicles, motor vehicles, road signs and landmark lines) and target attributes (including the target's relative position in the spatial coordinate system, movement speed and heading estimation results) in different scenes are obtained. The type of driving event (including overtaking, following, and collision risk) is set, and the hierarchical clustering method based on nearest neighbors is used to obtain the classification results of the driving event from the annotated data corresponding to the driver's visual behavior history data and the driving history data. Specifically including:

[0069] (1) Define the target i in a scene as node n i , with characteristic vector f i =[c,p x ,p y ,v x ,v y ] T , where c represents the target category (including pedestrians, non-motor vehicles, motor vehicles, road signs and landmark lines), p x and p y Indicates the lateral and longitudinal distance relative to the main vehicle (i.e. the vehicle driven by the driver), v x and v y Indicates the lateral and longitudinal speed relative to the host vehicle.

[0070] (2) Let the two targets in the scene form a motif pair, written as m = (n i ,n j ), satisfying i≠j, and using motif pairs to represent possible driving events between two targets. The feature of the motif is defined as the feature vector f i and f j The embedded combination of f m =[f i ,f j Given a dataset X N×d ={m1,m2,…,m N}, contains N motifs with a feature length of d.

[0071] (3) Calculate the nearest neighbor of each motif. The distance matrix of the dataset X is an N×N square matrix. D(m i ,m j ) is the motif m i and m j The Euclidean distance between:

[0072]

[0073] in, Motif m i The eigenvector of Motif m j The eigenvector of the motif m i The k nearest neighbors can be written as kNN(m i ):

[0074] kNN(m i )={m j |D(m i ,m j )≤D(m i ,NN k (m i ))}.

[0075] Among them, NN k (m i ) is the motif m i The kth nearest neighbor of . The shared neighbors between two motifs are used to represent their similarity, and the operator size(·) is used to calculate the number of elements in the set:

[0076] similarity(m i ,m j )=size(kNN(m i )∩kNN(m j )).

[0077] To express the separation of motifs, motif mi The k inverse nearest neighbors can be expressed as kRNN(m i ):

[0078] kRNN(m i )={m j |D(m i ,m j )≤D(m j ,NN k (m j ))}.

[0079] Where m i ∈NN k (m j ), we can get m j ∈kRNN(m i ). Motif m i The number of kRNNs is expressed as r(m i ):

[0080] r(m i )=|kRNN(m i )|.

[0081] Clusters are established through stability analysis, and the neighbor relationship is defined as:

[0082] NN1(m i )=m j ∧NN1(m j )=m i .

[0083] Among them, NN1(m i ) and NN1(m j ) are motif m i and m j The first nearest neighbor of two motifs m i and m j The stability between is defined as:

[0084]

[0085] Among them, kNN(m i ) and kNN(m j ) represents motif m i and m j The k nearest neighbors of m i and m j The stable connection relationship between them should meet the following conditions:

[0086] 1) If m i and m jare each other’s first nearest neighbors, then the connecting edge is bidirectional. The structure consists of the motif m i , m j and bidirectional connecting edges e ij It can be considered stable. Then, the motif m i and m j can be classified into the same cluster.

[0087] 2) If m i It is m j The first nearest neighbor of m j Not m i The first nearest neighbor of m i and m j The unidirectional edge of t cannot guarantee the stability of this structure. s Used to evaluate the stability of two motifs. If the shared neighbors of the two motifs are greater than th s ×k, then this unidirectional structure is stable and the connected motifs can be merged into the same cluster.

[0088] (4) Define two clusters C i and C j , and the connection represents R i|j Used to represent the cluster C i Relative to C j Relationship:

[0089]

[0090] C i and C j The motif pairs with k shared nearest neighbors in can be selected as the connection representatives of the two clusters respectively. i|j To ensure the number of representatives to avoid loss of accuracy. C i Relative to C j The extended representative set of is defined as follows:

[0091]

[0092] The extended connected representation is less restrictive than the original connected representation because it only requires the nearest neighbors. Then C i and C j The inter-cluster distance is defined as:

[0093]

[0094] Among them, R i =R i|j ∪R' i|j , R j =R j|i ∪R'j|i According to C i and C j The inter-cluster distance is defined as , and the nearest neighbors between clusters are used as representatives and the representative pairwise distance is calculated to measure the similarity between clusters. The nearest neighbor hierarchical clustering method continuously merges clusters with the smallest inter-cluster distance until the final clustering result is obtained.

[0095] (5) The final clustering result will be marked with the category of the driving event. In real-time calculation, the distance d (m i ,C m ), and the driving event category corresponding to the motif is determined by the minimum distance and cluster radius.

[0096] As an optional implementation, to improve the adaptability and generalization capabilities of the driver's target attention prediction model, a multi-layer visual memory environment model is integrated, including: a current scene layer, a historical sequence layer, and an attention trajectory layer. The current scene layer is derived based on the target recognition model and the driving event classification model; the historical sequence layer is derived from the current scene layer; and the attention trajectory layer is used to record the results of the current scene layer based on the historical sequence layer.

[0097] For example, the scene graph model at time t is defined as in and ε t They are the node set and the edge set respectively. Contains the target recognition results, i.e. The set of connected edges ε t Contains the driving event recognition results, indicating the potential interaction between nodes, namely, e ij ∈ε t The historical sequence layer is a continuous combination of the current scene layer in time sequence, defining the sequence and a window w, a subsequence G T-w+1:T =[G T-w+1 ,G T-w+2 ,…,G T ] constitutes a historical sequence. There are two motif structures in the historical sequence, which respectively represent the probability and relevance of the target in the history. They are calculated using the frequency-inverse document frequency method (TF-IDF) and point-wise mutual information (PMI). The calculation formula of the point-wise mutual information value is:

[0098]

[0099] Where p(i,j) is the probability that the graph contains both motif i and motif j, p(i) is the probability that the graph contains motif i, and p(j) is the probability that the graph contains motif j. The probabilities can be calculated as:

[0100]

[0101] Where N(i,j) represents the number of graphs containing both motif i and motif j, N(i) and N(j) represent the number of graphs containing motif i or motif j respectively, M is the total number of graphs, and motif node pairs with negative PMI values ​​are assigned zero weight. The calculation formula of TF-IDF is:

[0102]

[0103] Among them, B(i) j is the number of times motif or node i appears in graph j. The attention trajectory layer records the prediction results of the prediction model in each frame based on the historical sequence layer. The connecting edge represents the transfer path of attention. When the attention is transferred from target i to target j, the connecting edge e ij The value of is set to 1.

[0104] As an optional implementation, in order to improve the accuracy of the driver's target attention prediction model, the target-level visual attention prediction model includes: a multi-layer perceptron.

[0105] For example, a three-layer graph neural network (GNN) is used to learn the feature embedding results of visual memory knowledge. The GNN activation function is ReLU, and the COMBINE operator is used for feature embedding after linear mapping. A multi-layer perceptron (MLP) is used as a decoder to decode the encoded feature vector obtained by fusing the multi-layer visual memory environment model and generate a visual attention prediction target.

[0106] Based on the above embodiments proposed in this application, in actual applications, the driver's visual behavior data and driving data are obtained in real time; the current moment G is obtained through the target recognition model and the driving event recognition model. t ; G t It is used to construct the current scene layer and update the historical sequence layer. The prediction results at each moment in the updated historical sequence layer are used to update the attention trajectory layer. The updated environment model that integrates multiple layers of visual memory encodes the input and outputs the encoded feature vector; the target-level visual attention prediction model generates a visual attention prediction target based on the encoded feature vector.

[0107] Compared with the prior art, this application has the following advantages:

[0108] (1) Compared with the target attention prediction technology based on traditional machine learning, this application improves the adaptability to complex scenes and improves the accuracy and interpretability of the model prediction results. Machine learning-based methods often take raw sensor data as input, rely on the training effect of the data set, or manually extract underlying features (such as optical flow, color, etc.), which makes it difficult to describe the complex interactive relationships in traffic scenes. This application obtains the driving data of drivers in various driving scenarios, models the driving scene as a collection of independent targets and driving events, introduces the interaction between targets in the environmental model, can more accurately describe multi-target scenes, and improves the applicability to complex traffic scenes.

[0109] (2) Compared with the existing pixel-based target attention prediction technology, the present application uses a graphical model to output a unique predicted target, which improves the directionality of the prediction results in dynamic and complex scenes and avoids the situation where the probability map covers multiple targets at the same time and the driver's perception behavior is inconsistent. For example, when a driver turns in a crowded traffic scene, the multiple targets in the scene often have high observation weights due to factors such as category attributes, spatial positions, and motion characteristics. They are located in the high-probability area of ​​the output map, and the time-series distribution and transfer process of attention cannot be obtained. In order to avoid this situation, the present application combines the motif structure and the graphical model to model visual attention, so that attention has the ability to transfer at the temporal level, describes the influence of targets at different times on the current attention allocation, and the target-level visual attention prediction model processes the encoded feature vector and outputs a unique visual attention prediction target, thereby improving the accuracy of the prediction results in multi-target scenes.

[0110] Based on the same inventive concept, the present application also provides a driver target attention system for applying the aforementioned driver target attention method. The implementation solution provided by this system is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of the driver target attention system embodiment provided below can be found in the above-mentioned limitations of the driver target attention method and will not be repeated here.

[0111] In an exemplary embodiment, Figure 2 As shown, a driver target attention system is provided, comprising:

[0112] The driving data collection module 1 is used to collect the driver's visual behavior data and driving data in real time.

[0113] The target recognition module 2 is used to process the driver's visual behavior data and driving data, and combine the information in the on-board camera image and the lidar point cloud to obtain the target recognition result.

[0114] Driving event classification module 3 is used to process driving data based on the target recognition results to obtain driving event classification results. It uses the nearest neighbor hierarchical clustering method to obtain driving event clusters and to identify the driving event categories corresponding to the potential relationships between two targets in the target recognition results in real time.

[0115] The attention prediction module 4 is configured to obtain a visual attention prediction target based on the target recognition result and the driving event classification result, and to predict the future driver's visual attention distribution target in real time.

[0116] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data used in the driver's target attention prediction method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a driver's target attention method is implemented.

[0117] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0118] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0119] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0120] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0122] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0124] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0125] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for predicting driver's target attention, characterized in that: The driver's target attention prediction method includes: Constructing a driver's target attention prediction model; the driver's target attention prediction model includes: a target recognition model, a driving event classification model and an attention prediction model; the attention prediction model includes an environment model integrated with multi-layer visual memory and a target-level visual attention prediction model; the target recognition model is used to obtain a target recognition result based on the driver's visual behavior data and driving data; the driving event classification model is used to obtain a driving event classification result based on the target recognition result and the driving data corresponding to the target recognition result; the environment model integrated with multi-layer visual memory is used to obtain a coding feature vector based on the target recognition result and the driving event classification result; the target-level visual attention prediction model is used to obtain a visual attention prediction target based on the coding feature vector; Acquire driver's visual behavior data and driving data in real time; The driver's visual behavior data and the driving data are input into the driver's target attention prediction model to obtain the visual attention prediction target.

2. The method for predicting driver's target attention according to claim 1, characterized in that: The constructing of the driver's target attention prediction model includes: Constructing a first sample data set and the target recognition model; the first sample data set includes driver visual behavior history data, driving history data, and annotated data corresponding to the driver visual behavior history data and the driving history data; the driver visual behavior history data includes: first-person perspective video, binocular iris positioning, and viewpoint projection coordinates; the driving history data includes: lidar point cloud and vehicle-mounted camera images; Using the first sample data set to train and test the target recognition model until a first set condition is met, and using the trained target recognition model as a trained target recognition model; Constructing a second sample data set and the driving event classification model; the second sample data set includes annotated data corresponding to the driver's visual behavior history data and the driving history data; using the second sample data set to train and test the driving event classification model until a second set condition is met, and using the trained driving event classification model as a trained driving event classification model; Constructing a third sample data set and the attention prediction model; the third sample data set includes annotated data corresponding to the driver's visual behavior history data and the driving history data; the attention prediction model includes the environment model fused with multi-layer visual memory and the target-level visual attention prediction model; The attention prediction model is trained and tested using the third sample data set until a third set condition is met, and the trained attention prediction model is used as the trained attention prediction model.

3. The method for predicting driver's target attention according to claim 2, characterized in that: The constructing of the first sample data set includes: Obtaining, based on the driver's visual behavior history data and the driving history data, labeled data corresponding to the driver's visual behavior history data and the driving history data to generate a training sample pair; A first sample data set is constructed based on the training sample pairs.

4. The method for predicting driver's target attention according to claim 2, characterized in that: Constructing the driving event classification model includes: using a hierarchical clustering method based on nearest neighbor to obtain a classification result of the driving event according to the labeled data corresponding to the driver's visual behavior history data and the driving history data.

5. The method for predicting driver's target attention according to claim 1, characterized in that: The fused multi-layer visual memory environment model includes: a current scene layer, a history sequence layer, and an attention trajectory layer; Obtaining the current scene layer based on the target recognition model and the driving event classification model; obtaining the historical sequence layer based on the current scene layer; The attention trajectory layer is used to record the results of the current scene layer based on the historical sequence layer.

6. The method for predicting driver's target attention according to claim 1, characterized in that: The target-level visual attention prediction model includes: a multi-layer perceptron; The multi-layer perceptron is used to decode the encoded feature vector and generate the visual attention prediction target.

7. A driver's target attention prediction system, characterized in that: The driver target attention prediction system includes: Driving data collection module, used to collect driver visual behavior data and driving data in real time; a target recognition module, configured to process the driver's visual behavior data and the driving data to obtain a target recognition result; a driving event classification module, configured to process the driving data based on the target recognition result to obtain a driving event classification result; The attention prediction module is used to obtain a visual attention prediction target based on the target recognition result and the driving event classification result.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the driver target attention prediction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the driver target attention prediction method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the driver target attention prediction method according to any one of claims 1 to 6 is implemented.

Images