Driver attention prediction method, system, device, storage medium and product

Abstract

The application discloses a driver attention prediction method, system, device, storage medium and product, belongs to the technical field of intelligent automobile driver attention prediction methods, and solves the problem that the prior art still has defects in accuracy and is easily disturbed by noise, leading to unstable model output. A random Gaussian noise image is generated; a motion pattern image is generated; the random Gaussian noise image is reversely denoised to generate a reverse random Gaussian noise image; a traffic scene RGB image corresponding to a driver attention heat map, the motion pattern image and the reverse Gaussian noise image are guided by conditional information to generate a spliced noise image; the spliced noise image and the corresponding driver attention heat map are input into a conditional denoising diffusion probability model for training to obtain a final conditional denoising diffusion probability model; and a to-be-predicted traffic scene RGB image is input into the final conditional denoising diffusion probability model to generate a driver attention prediction result, namely a driver attention heat map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent automobile driver attention prediction methods, in particular to a driver attention prediction method, system, device, storage medium and product. BACKGROUND

[0002] In recent years, with the continuous development and progress of various technologies of intelligent automobiles, it is helpful for intelligent automobiles to better understand the traffic environment and thus more efficiently make subsequent decision planning by letting the intelligent automobile system imitate and learn the driving attention perspective of skilled drivers and realizing driver attention prediction. The goal of driver attention prediction is to predict the possible attention area of the driver in the current traffic scene image based on the driver's perspective of the traffic scene image at the current time and in the past. Since the traffic environment is a complex and dynamic scene, the driver must pay close attention to the significant and important targets or areas to drive safely, and the attention of skilled drivers contains the wisdom of human beings to filter out the trivial and extract the essence of various factors in the dynamic and complex traffic environment through systematic training and practical accumulation. Therefore, researching the visual attention distribution of the driver in the traffic scene and accurately reproducing the visual focus of the driver in the driving process to realize driver attention prediction are helpful for implanting human cognitive ability in intelligent automobiles and improving the safety and driving experience of autonomous driving to realize the human-machine fusion intelligence of intelligent automobiles.

[0003] The current driver attention prediction method is realized through graph convolution network, long short-term memory network, encoder-decoder network, Transformer network (attention mechanism network) and conditional adversarial generative network, etc. For example, Chinese patent CN115457521A discloses a "lightweight driver attention prediction method and device based on multi-layer spatio-temporal fusion", which designs a lightweight backbone and a lightweight 3D module through a multi-layer spatio-temporal fusion lightweight driver attention prediction method, and realizes driver attention prediction through an encoder and a decoder module. This method ensures the speed of driver attention prediction but sacrifices certain accuracy, and the method is not stable enough, resulting in insufficient accuracy. SUMMARY

[0004] The present application solves the problem of instability of the model output caused by the noise interference in the accuracy of the prior art.

[0005] The driver attention prediction method disclosed by the present application comprises the following steps:

[0006] Step S1, respectively acquiring a driver attention heat map and a traffic scene RGB image corresponding thereto;

[0007] Step S2, performing forward diffusion on the driver attention heat map to generate a random Gaussian noise image;

[0008] Step S3, the driver attention heat map corresponding traffic scene RGB image processing, generating motion pattern map;

[0009] Step S4, inverse random Gaussian noise image denoising, generating inverse random Gaussian noise image;

[0010] Step S5, the driver attention heat map corresponding traffic scene RGB image, motion pattern map and inverse Gaussian noise image are guided by conditional information, and the spliced noise image is generated;

[0011] Step S6, the spliced noise image and its corresponding driver attention heat map are input into the conditional denoising diffusion probability model for training, and the final conditional denoising diffusion probability model is obtained;

[0012] Step S7, the traffic scene RGB image to be predicted is input into the final conditional denoising diffusion probability model, and the driver attention prediction result driver attention heat map is generated.

[0013] Further, in the embodiment of the present application, in step S2, the driver attention heat map is diffused forward to generate a random Gaussian noise image, specifically:

[0014] The forward Markov chain is defined, and the Gaussian noise is gradually introduced into the driver attention heat map according to the variance sequence through multiple iterations to generate a random Gaussian noise image.

[0015] Further, in the embodiment of the present application, in step S3, the driver attention heat map corresponding traffic scene RGB image processing, generating light flow map.

[0016] Further, in the embodiment of the present application, the Farnebäck method is used to process the driver attention heat map corresponding traffic scene RGB image to generate a light flow map, specifically:

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] wherein, is the intensity of the image region, for pixel coordinates, and are both symmetric quadratic coefficient matrices, and are different linear coefficient vectors, and are different constant terms, is a transpose, is a pixel displacement vector, is time, and are different frame image region intensities, is a linear coefficient vector, is a constant term value, is a quadratic term coefficient matrix value, is an image intensity of time

[0024] Further, in the embodiment of the present application, the conditional information guidance in step S5 includes channel splicing.

[0025] Further, in the embodiment of the present application, the conditional denoising diffusion probability model in step S6 includes a U-Net model.

[0026] The driver attention prediction system provided by the present application, the system is realized by the driver attention prediction method of the above method, including the following modules:

[0027] An acquisition module acquires a driver attention heat map and a traffic scene RGB image corresponding to the driver attention heat map, respectively;

[0028] A forward diffusion module generates a random Gaussian noise image by performing forward diffusion on the driver attention heat map;

[0029] A processing module generates a motion pattern image by processing the traffic scene RGB image corresponding to the driver attention heat map;

[0030] A reverse denoising module generates a reverse random Gaussian noise image by performing reverse denoising on the random Gaussian noise image;

[0031] A conditional information guidance module generates a spliced noise image by performing conditional information guidance on the traffic scene RGB image corresponding to the driver attention heat map, the motion pattern image, and the reverse Gaussian noise image;

[0032] A training module trains the spliced noise image in a noise prediction network to obtain a final noise prediction network;

[0033] A generation module inputs a to-be-predicted traffic scene RGB image into the final noise prediction network to generate a driver attention prediction result, i.e., a driver attention heat map.

[0034] An electronic device according to the present application comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0035] The memory is used for storing a computer program.

[0036] The processor is used for executing the program stored on the memory to realize the driver attention prediction method according to any one of the above methods.

[0037] A computer readable storage medium according to the present application stores a computer program, and the computer program is executed by a processor to realize the driver attention prediction method according to any one of the above methods.

[0038] A computer program product according to the present application comprises a computer program or instructions, and the computer program or instructions are executed by a processor to realize the driver attention prediction method according to any one of the above methods.

[0039] The present application solves the problem of instability of the model output caused by noise interference in the prior art, and has the following beneficial effects:

[0040] 1. The driver attention prediction method according to the present application has higher generation stability and convergence. Compared with the existing generation adversarial network and Transformer network, the conditional denoising diffusion probability model used in the present application gradually adds and removes noise through Markov chain, avoiding the instability problem of adversarial training. The existing technology often causes large output fluctuations and difficult training convergence due to mode collapse (single generated sample) and unbalanced discriminator design. The present application significantly improves the convergence and training stability of the model by deterministic iterative optimization, and ensures that the generation process is more controllable and reliable.

[0041] 2. The driver attention prediction method according to the present application has stronger anti-noise interference ability. The existing method based on LSTM (Long Short-Term Memory Recurrent Neural Network) or graph convolution is sensitive to input noise and is easily affected by factors such as light change and occlusion, resulting in prediction deviation. The present application gradually injects noise through the forward diffusion process, so that the model learns the noise distribution during training, thereby effectively filtering out interference during generation. At the same time, the conditional information guide directly guides the denoising process through channel splicing, further enhancing the robustness of the model to noise, so that it can still maintain high-precision prediction in complex traffic scenes.

[0042] 3、The driver attention prediction method provided by the application has more accurate spatio-temporal feature fusion, the prior art usually only relies on static images or simple time sequence modeling, and it is difficult to comprehensively capture the spatio-temporal correlation of a dynamic traffic scene, the application introduces a double-condition guiding mechanism: an RGB image provides static spatial information, and an optical flow map captures dynamic motion features (such as speed and direction), through the skip connection and residual block design of the U-Net, the model realizes efficient fusion of multi-scale spatio-temporal features in the decoder, and the generated driver attention heat map is more in line with the visual attention distribution of a real driver;

[0043] 4、The driver attention prediction method provided by the application has higher generation quality and diversity, the existing generative adversarial network is prone to generate blurred or repetitive samples (mode collapse), and the lightweight model usually sacrifices accuracy to obtain speed, the application realizes iterative refinement generation, introduces conditional information fine-tuning at each step, ensures that the generated driver attention heat map is rich in details and has stronger diversity, and in addition, the optimized U-Net structure balances between calculation efficiency and feature extraction capability, avoids the accuracy loss of the lightweight model, and thus realizes double improvement in generation quality and efficiency;

[0044] The driver attention prediction method provided by the application introduces a conditional denoising diffusion probability model into the field of driver attention prediction, realizes high stability and high-precision driver attention heat map generation through a Markov chain process of forward diffusion and reverse denoising. BRIEF DESCRIPTION OF DRAWINGS

[0045] The above and / or additional aspects and advantages of the application will become apparent and be readily understood from the following description, taken in conjunction with the drawings, in which:

[0046] Figure 1 is a flow chart of the driver attention prediction method according to embodiment one. DETAILED DESCRIPTION

[0047] The various embodiments of the application will be described in detail below with reference to the drawings. The embodiments described by reference to the drawings are exemplary and are intended to explain the application, and cannot be understood as limiting the application.

[0048] Embodiment one, the driver attention prediction method according to the embodiment includes the following steps:

[0049] Step S1, respectively acquire a driver attention heat map and a traffic scene RGB image corresponding to the driver attention heat map;

[0050] Step S2, forward diffuse the driver attention heat map to generate a random Gaussian noise image;

[0051] Step S3, the driver attention heat map corresponding traffic scene RGB image processing, generating motion pattern map;

[0052] Step S4, inverse denoising of random Gaussian noise image, generating inverse random Gaussian noise image;

[0053] Step S5, the driver attention heat map corresponding traffic scene RGB image, motion pattern map and inverse Gaussian noise image are guided by conditional information, generating spliced noise image;

[0054] Step S6, the spliced noise image and its corresponding driver attention heat map input conditional denoising diffusion probability model for training, obtaining the final conditional denoising diffusion probability model;

[0055] Step S7, the traffic scene RGB image to be predicted input final conditional denoising diffusion probability model, generating driver attention prediction result driver attention heat map.

[0056] In the embodiment, in step S2, the driver attention heat map is forward diffused to generate a random Gaussian noise image, specifically:

[0057] The forward Markov chain is defined, and by multiple iterations, the Gaussian noise is gradually introduced into the driver attention heat map according to the variance sequence to generate a random Gaussian noise image.

[0058] In the embodiment, in step S3, the driver attention heat map corresponding traffic scene RGB image processing, generating light flow map.

[0059] In the embodiment, the Farnebäck method is used to process the driver attention heat map corresponding traffic scene RGB image to generate a light flow map, specifically:

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] wherein, is the intensity of the image region, is the pixel coordinate, and are symmetric quadratic coefficient matrices, and are different linear coefficient vectors, and are different constant terms, is a transpose, is a pixel displacement vector, is time, and are different frame image region intensities, is a linear coefficient vector, is a constant term value, is a quadratic term coefficient matrix value, is the image intensity of time .

[0067] In the embodiment, the conditional information guidance in step S5 includes channel splicing.

[0068] In the embodiment, the conditional denoising diffusion probability model in step S6 includes a U-Net model.

[0069] The prior art still has deficiencies in accuracy and is easily disturbed by noise, resulting in unstable model output. In particular, the conditional generative adversarial network still has problems of unstable adversarial mechanism, unbalanced discriminator design and training, and limited pattern collapse of output sample types. At the same time, the accuracy of model prediction needs to be improved.

[0070] Therefore, in order to further improve the effect of driver attention prediction, the embodiment explores more stable and diversified methods, such as Figure 1 As shown in the figure, a driver attention prediction method based on a conditional denoising diffusion probability model is proposed, which includes the following steps:

[0071] Step S1, respectively acquire the driver attention heat map and the traffic scene RGB graph corresponding thereto, specifically:

[0072] A driver attention grayscale image and a corresponding traffic scene RGB image (red, green, and blue image) are obtained and preprocessed to establish a dataset. The driver attention grayscale image is a grayscale image formed by the driver's visual attention area changing with the scene when the driver views the traffic scene front view image collected by an eye tracker device. Each pixel point has a grayscale value of 0-255, and a higher grayscale value of a region indicates that the driver pays more attention to the region. Since the driver attention grayscale image can be converted into a driver attention heat map to enhance the visualization effect through a color mapping method, the two images are equivalent, and are collectively referred to as a driver attention heat map below. The data set is obtained from a public data set or a self-collected data set. The driver attention heat map and the corresponding traffic scene RGB image are converted into 256x256 pixel size through scaling transformation for subsequent steps.

[0073] In step S2, the driver attention heat map is forward diffused to generate a random Gaussian noise image, specifically:

[0074] First, a forward Markov chain is defined Through times of iteration, the noise variance sequence is gradually introduced into the driver attention heat map, until it becomes a random Gaussian noise image ,

[0075] ; (1)

[0076] ; (2)

[0077] wherein is the diffusion time step, the noise variance is a hyperparameter that determines the Gaussian noise variance introduced at each iteration, which quantifies the noise intensity at each step. In the forward diffusion process, increases from small to large, so that the noise gradually increases with the increase of the diffusion time step , while avoiding introducing too much noise in the initial stage, is a key parameter that determines the noise intensity at each step, which is linearly transformed with the diffusion time step , so that the noise intensity is also linearly adjusted to adapt to different diffusion time steps , is the forward Markov chain, is the driver attention heat map original image , the partial noise image after adding noise to the driver attention heat map original image, is the partial noise image of the previous step, is the unit covariance matrix, is the driver attention heat map original image.

[0078] Let , , At any diffusion time step sample:

[0079] ; (3)

[0080] For simplicity of notation, let , the above equation can be expressed as:

[0081] ; (4)

[0082] where decreases as increases to ensure that the variance of the random variable remains bounded at After reparameterization, we obtain the random Gaussian noise image :

[0083] ; (5)

[0084] where is a standard Gaussian noise, following a standard Gaussian distribution.

[0085] Step S3, the driver attention heat map corresponding to the traffic scene RGB image is processed to generate a motion pattern image, specifically:

[0086] The driver attention heat map corresponding to the traffic scene RGB image is processed to generate a motion pattern image, and the motion pattern image includes a time sequence image of an optical flow image and a non-optical flow image. The optical flow image is used to describe the movement of pixels in an image sequence, which can capture the movement information of objects, backgrounds or cameras between consecutive frames, thereby revealing the motion pattern of each part in the picture. The core idea of the optical flow image is to estimate the motion direction and speed of pixels by tracking their positions at different time points. The Farnebäck method (optical flow method) is used to generate the optical flow image, which is expressed as follows:

[0087] ; (6)

[0088] ; (7)

[0089] ; (8)

[0090] ; (9)

[0091] ; (10)

[0092] ; (11)

[0093] wherein, is the image region intensity, is the pixel coordinate, and are symmetric quadratic coefficient matrices, and are different linear coefficient vectors, and are different constant terms, is the transpose, is the pixel displacement vector, is the time, and are different frame image region intensities, is the linear coefficient vector, is the constant term value, is the quadratic term coefficient matrix value, is the image intensity at time .

[0094] Step S4, inverse de-noising is performed on the random Gaussian noise image to generate an inverse random Gaussian noise image, specifically:

[0095] In the inverse de-noising process, the inference from the random Gaussian noise image to the driver attention heat map is defined as an inverse Markov process, which is contrary to the forward diffusion process. The inference gradually de-noises from the random Gaussian noise image :

[0096] ; (12)

[0097] ; (13)

[0098] ; (14)

[0099] wherein, is the traffic scene condition information image, is the model parameter, and the inference process is defined from the random Gaussian noise image , through an inverse de-noising process, to finally obtain the target image driver attention heat map . This inference process includes an iterative refinement step, and each step uses a conditional distribution , which is learned by a neural network model . If the variance of the noise in the forward diffusion process is set to be small enough (i.e. ), the optimal inverse process This approximates a Gaussian distribution. Therefore, choosing a Gaussian conditional distribution during the reasoning process can provide a reasonable fit. Meanwhile, in equation (4)... Large enough to guarantee random Gaussian noise images Conforms to prior distribution This causes the sampling process to begin with completely random Gaussian noise. The standard deviation of the sampling is . The image contains random Gaussian noise. This represents the predicted mean of the neural network model. It is a reverse Markov process.

[0100] From equation (8), we can further define the following equation:

[0101] (15)

[0102] (16)

[0103] in, The conditional distribution mean term, This is the variance adjustment factor. and The scaling factor for cumulatively retaining the original image information.

[0104] The reverse denoising process is further parameterized as follows:

[0105] (17)

[0106] (18)

[0107] in, This represents the predicted mean of the neural network model. Input information based on conditions. A scaling factor to preserve the original image information. and This is a condition-guided noise control factor.

[0108] The key to the reverse denoising process is training a neural network model. This model can be applied to images with random Gaussian noise. and diffusion time step Predict noise levels under certain conditions. During training, a neural network model is used. To predict noise Therefore, given a random Gaussian noise image By using By replacing and rearranging the items, a heatmap of driver attention approximating the target image is obtained. The expression:

[0109] (19)

[0110] wherein, is the estimated driver attention heat map image.

[0111] The mean of can be parameterized as:

[0112] (20)

[0113] wherein, and are variance parameters of injected noise in the forward diffusion process. After this parameterization, each iteration step is represented as:

[0114] (21)

[0115] wherein, is a standard Gaussian noise.

[0116] In the reverse denoising process of step , the traffic scene condition information image is applied to each step, and when it is taken as an input condition, it can provide more information for the neural network model, which reflects the structural and contour features of the driver attention heat map , effectively providing prior knowledge for the neural network model, which helps the neural network model accurately predict noise and restore the original image, and the neural network model can more effectively utilize the changes in noise variance to help accurately predict noise, thereby improving the effectiveness and quality of the generated driver attention heat map.

[0117] Step S5, the traffic scene RGB image corresponding to the driver attention heat map, the motion pattern image and the reverse Gaussian noise image are guided by the condition information to generate a spliced noise image, specifically:

[0118] In the condition information guiding part, the traffic scene RGB image corresponding to the driver attention heat map is taken as the main input data, which provides rich visual information of the traffic scene, including roads, traffic signs, driving vehicles and other key elements. The motion pattern image in this embodiment uses an optical flow map, which provides motion information of each pixel point in the traffic scene, reflecting the motion speed and direction of objects in the image relative to the driver, which is crucial for capturing the dynamic changes of the driver's current attention area. The traffic scene RGB image corresponding to the driver attention heat map and the optical flow map are combined to generate a condition information image, which is then guided by the condition information with the reverse Gaussian noise image to generate a spliced noise image.

[0119] The conditional information guidance is realized by channel concatenation, which is a widely used feature fusion method. The core idea is to concatenate multiple feature tensors along the channel dimension to integrate feature information from different sources or modules. Specifically, the input inverse Gaussian noise image and the conditional information image feature tensor are and wherein, is the real space of the range of tensor values, and represent the height and width of the feature, respectively, and represent the number of channels of the feature, respectively. Through the channel concatenation operation, a new feature tensor This operation not only preserves the spatial resolution of the original feature, but also effectively integrates the conditional information and the main feature, thereby providing more rich feature support for the driver attention heat map generation process. At the same time, the channel concatenation operation has high computational efficiency and can seamlessly integrate multiple information, so it becomes an intuitive and efficient guidance method for conditional generation tasks.

[0120] The channel concatenation method of conditional information guidance used in this embodiment can also be replaced by Cross-Attention (cross-attention). Insert a cross-attention layer in the U-Net to make the noise prediction network dynamically focus on the key areas of the conditional information.

[0121] Therefore, the embodiment realizes spatio-temporal information fusion through conditional information guidance. Spatial information provides static environmental features (such as road structure, vehicle position) through traffic scene RGB images. Temporal information captures dynamic motion features (such as vehicle speed, direction) through optical flow maps. Through channel concatenation, both are input into the model as conditional information to guide the inverse denoising process and enhance the relevance of the driver attention heat map generation result to the traffic scene.

[0122] Step S6, the concatenated noise image and its corresponding driver attention heat map are input into the conditional denoising diffusion probability model for training to obtain the final conditional denoising diffusion probability model, specifically:

[0123] Combining the traffic scene RGB image corresponding to the driver attention heat map as the conditional guidance information with the conditional denoising diffusion probability model enables the model to generate a more accurate attention distribution map according to the visual features of the current scene. The traffic scene RGB image corresponding to the driver attention heat map guides the denoising process to ensure that the generated attention distribution map fully extracts spatial information.

[0124] The optical flow map is taken as another conditional information input, combined with the traffic scene RGB image corresponding to the driver attention heat map, so that the model can consider the motion timing information in the scene, and thus better capture the driver's attention distribution in the traffic scene. Through the guidance method of the two kinds of conditional information, the static spatial information and dynamic timing information can be considered comprehensively when generating the driver attention heat map, thereby improving the accuracy and robustness of attention prediction.

[0125] Therefore, the embodiment adopts traffic scene RGB images on spatial information and optical flow maps on timing information, and performs channel splicing on the traffic scene RGB images, the optical flow maps and the inverse Gaussian noise images to simultaneously input the inverse Gaussian noise images as conditions to the denoising diffusion probability model for noise prediction, thereby realizing the fusion of spatial and timing feature information.

[0126] In step S4, in the inverse denoising process, a neural network model is used to predict noise at each step , the input of the neural network model includes a random Gaussian noise image and a diffusion time step , and the output is a noise prediction . In the training process, the neural network model is only used to predict noise once, while in the inverse denoising process, iterations are required, which is usually set to about 1000-2000. In each iteration, the neural network model is used to predict noise, and a deterministic formula is used to update the random Gaussian noise image . Although the amount of calculation during training is not large, since the process needs to be repeated times, it may become time-consuming and increase the computational burden during testing.

[0127] ​​​To reduce the computational burden, the conditional denoising diffusion probability model described in the embodiment generally uses a relatively simple U-Net model (an image segmentation convolutional neural network model) that effectively combines local and global features through a symmetrical encoding and decoding structure and simultaneously passes detailed information in the image through a skip connection, thus performing well in the fields of medical images, semantic segmentation, and super-resolution. The conditional denoising diffusion probability model uses a U-Net model containing 3 ResNets (residual blocks), sets the channel number of the first layer to 64, and the depth multiplier to {1, 2, 4, 8, 16}. In the encoder downsampling part, the encoder part is similar to a traditional convolutional neural network, which extracts image features through a series of convolutional layers and pooling layers. As the network deepens, the spatial size of the feature map gradually decreases, while the number of channels gradually increases, thereby capturing more image features that may represent key areas in the traffic scene, such as vehicles, road signs, etc. This information helps to predict the driver's attention distribution. In the decoder upsampling part, the decoder upsamples the feature map to restore the spatial resolution of the image, and uses deconvolution upsampling operations to restore the image to its original size. At the same time, skip connections are introduced between the encoder and the decoder to directly pass the features of each layer in the encoder to the corresponding layer in the decoder. These skip connections enable effective features to directly participate in the generation process of the attention distribution map, which helps to more accurately predict the driver's attention distribution.

[0128] The conditional denoising diffusion probability model used in the embodiment is a U-Net model, which can also be replaced by other conditional denoising diffusion probability models, such as ViT (Vision Transformer, a way of processing images in the field of computer vision) that captures global spatiotemporal dependencies through multi-head attention mechanisms, GNN (Graph Neural Network) fusion, etc.

[0129] Based on the above steps, the conditional denoising diffusion probability model is trained, and the loss function for model training is designed. Since the traffic scene conditional information image is introduced to achieve controllable image generation as additional condition information, and the neural network model is optimized in the reverse denoising process , the function takes the traffic scene conditional information image , a random Gaussian noise image , and a diffusion time step as input to predict noise Therefore, the loss function can be represented as:

[0130] (22)

[0131] wherein , , L1 norm, loss function, mathematical expectation over all training samples, joint expectation of noise and time step, noise, mathematical expectation, training sample.

[0132] By integrating the conditional information guidance and the conditional denoising diffusion probability model into the reverse denoising process, effective and accurate prediction of the driver attention heat map is achieved. The model is trained until the model converges.

[0133] Therefore, compared with existing generative adversarial networks and Transformer networks, the use of the conditional denoising diffusion probability model avoids mode collapse and unstable adversarial training through progressive generation, and outputs more stable and diversified results.

[0134] Step S7, the traffic scene RGB image to be predicted is input into the final conditional denoising diffusion probability model to generate a driver attention prediction result, i.e., a driver attention heat map, specifically as follows:

[0135] The current time traffic scene RGB image to be predicted and the previous time traffic scene RGB image are input, a flow graph is generated according to the flow graph generation method in step S3, and then the current time traffic scene RGB image to be predicted and the flow graph are input as conditional guidance information into the trained final conditional denoising diffusion probability model, and denoising iteration is continuously performed until the current time driver attention prediction result, i.e., the driver attention heat map, is output.

[0136] Therefore, the present embodiment realizes driver attention prediction by introducing conditional information guidance based on the original conditional denoising diffusion probability model. In the forward diffusion process, a Markov decision process is introduced, and Gaussian noise is gradually added to the driver attention prediction graph until it becomes a random noise image. In the reverse denoising process, an inverse Markov decision process is realized, and through a U-Net encoder-decoder neural network containing three Resnets, the random noise image is gradually denoised into a driver attention prediction graph. At the same time, in the reverse denoising process, the spatiotemporal features are fused, and the conditional guidance information is introduced, i.e., the traffic scene RGB image spatial information and the flow graph temporal information are used as conditional information guidance, and through the random iterative denoising process, the corresponding driver attention heat map is generated from the random noise image.

[0137] The method has higher stability and better convergence performance, can flexibly cope with changes in the driver's attention focus area in different scenes by introducing conditional information guidance, can effectively guide the model to generate a driver attention prediction result attention heat map that conforms to the actual scene, thereby improving the relevance of the generated image and the accuracy of the prediction task.

[0138] Embodiment two, the driver attention prediction system described in the embodiment, the system is realized by the driver attention prediction method described in embodiment one, comprising the following modules:

[0139] An acquisition module acquires a driver attention heat map and a traffic scene RGB image corresponding thereto, respectively;

[0140] A forward diffusion module diffuses the driver attention heat map forward to generate a random Gaussian noise image;

[0141] A processing module processes the traffic scene RGB image corresponding to the driver attention heat map to generate a motion pattern image;

[0142] A reverse denoising module denoises the random Gaussian noise image in reverse to generate a reverse random Gaussian noise image;

[0143] A conditional information guidance module guides the traffic scene RGB image corresponding to the driver attention heat map, the motion pattern image, and the reverse Gaussian noise image to generate a spliced noise image;

[0144] A training module inputs the spliced noise image and the corresponding driver attention heat map into a noise prediction network for training to obtain a final noise prediction network;

[0145] A generation module inputs a to-be-predicted traffic scene RGB image into the final noise prediction network to generate a driver attention prediction result driver attention heat map.

[0146] Embodiment three, an electronic device described in the embodiment, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, the memory complete mutual communication through the communication bus;

[0147] The memory is used to store a computer program;

[0148] The processor is used to execute the program stored on the memory to realize the driver attention prediction method described in embodiment one.

[0149] Embodiment four, a computer readable storage medium described in the embodiment, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the driver attention prediction method described in embodiment one.

[0150] Embodiment five, the computer program product includes computer programs or instructions, which are executed by a processor to implement the driver attention prediction method of embodiment one.

[0151] The driver attention prediction method, system, device, storage medium and product of the present application are described in detail above. The principles and embodiments of the present application are described by applying specific examples. The above examples are only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific embodiments and application scope can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for predicting driver attention, characterized in that, Includes the following steps: Step S1: Obtain the driver's attention heatmap and the corresponding traffic scene RGB image; Step S2: Forwardly diffuse the driver attention heatmap to generate a random Gaussian noise image; Step S3: Process the RGB image of the traffic scene corresponding to the driver attention heatmap to generate a motion pattern map; Step S4: The random Gaussian noise image is reversed to generate an inverse random Gaussian noise image; Step S5: Conditional information is used to guide the generation of a stitched noise image by combining the RGB image of the traffic scene, the motion pattern image, and the inverse Gaussian noise image corresponding to the driver attention heatmap. Step S6: The stitched noisy image and its corresponding driver attention heatmap are input into the conditional denoising diffusion probability model for training, to obtain the final conditional denoising diffusion probability model. Step S7: Input the RGB image of the traffic scene to be predicted into the final conditional denoising diffusion probability model to generate the driver attention prediction result, driver attention heatmap. In step S3, the RGB image of the traffic scene corresponding to the driver attention heatmap is processed to generate an optical flow map.

2. The driver attention prediction method according to claim 1, characterized in that, In step S2, the forward diffusion of the driver attention heatmap to generate a random Gaussian noise image specifically involves: Define a forward Markov chain and gradually introduce Gaussian noise into the driver attention heatmap according to the variance sequence through multiple iterations to generate a random Gaussian noise image.

3. The driver attention prediction method according to claim 1, characterized in that, The Farnebäck method is used to process the RGB image of the traffic scene corresponding to the driver attention heatmap to generate an optical flow map, specifically: ； ； ； ； ； ； in, For image region intensity, For pixel coordinates, and Both are symmetric quadratic coefficient matrices. and These are vectors of different linear coefficients. and These are different constant terms, For transpose, For pixel displacement vectors, For time, and The intensities of different frame image regions are respectively. A linear coefficient vector, For the value of the constant term, The values ​​of the quadratic coefficient matrix are... For time Image intensity.

4. The driver attention prediction method according to claim 1, characterized in that, In step S5, the conditional information guidance includes channel splicing.

5. The driver attention prediction method according to claim 1, characterized in that, In step S6, the conditional denoising diffusion probability model includes the U-Net model.

6. A driver attention prediction system, wherein the system is implemented using the driver attention prediction method of claim 1, characterized in that, Includes the following modules: The acquisition module acquires a heatmap of the driver's attention and a corresponding RGB image of the traffic scene. The forward diffusion module forward diffuses the driver attention heatmap to generate a random Gaussian noise image; The processing module processes the RGB image of the traffic scene corresponding to the driver attention heatmap to generate a motion pattern map; The inverse denoising module performs inverse denoising on the random Gaussian noise image to generate an inverse random Gaussian noise image; The conditional information guidance module guides the driver's attention heatmap by applying conditional information to the traffic scene RGB image, motion pattern image and inverse Gaussian noise image corresponding to the driver's attention heatmap, and generates a stitched noise image. The training module inputs the stitched noisy image into the noise prediction network for training, and obtains the final noise prediction network. The generation module takes the RGB image of the traffic scene to be predicted as input into the final noise prediction network and generates a driver attention heatmap as the driver attention prediction result.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When executing a program stored in memory, the processor implements the driver attention prediction method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the driver attention prediction method according to any one of claims 1-5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the driver attention prediction method according to any one of claims 1-5.

Images