Traffic video saliency prediction model generation method and traffic video saliency prediction method

Abstract

The present disclosure relates to a traffic video saliency prediction model generation method and a traffic video saliency prediction method. The traffic video saliency prediction model generation method comprises: obtaining a traffic video training sample set; the traffic video training sample set comprises a plurality of traffic video clips and a saliency heat map of each video frame; wherein each traffic video clip comprises a plurality of continuous video frames; constructing an initial video saliency prediction model; the initial video saliency prediction model is composed of a priori knowledge learning subnetwork and a video saliency prediction subnetwork sharing an encoder; based on the traffic video training sample set, the priori knowledge learning subnetwork is pre-trained to learn traffic video priori knowledge, and a preset video saliency prediction model is obtained; based on the learned traffic video priori knowledge, the video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned to obtain a target video saliency prediction model.

Description

Technical Field

[0001] This disclosure relates to the field of digital image processing vision, and in particular to a method for generating a traffic video saliency prediction model and a method for predicting traffic video saliency. Background Technology

[0002] Video saliency prediction, which predicts the direction of human gaze, can improve the efficiency of traffic surveillance video processing, thereby increasing the efficiency of extracting key information from massive amounts of video. Traffic surveillance videos differ significantly from ordinary scene videos. Traffic surveillance encompasses various scenarios, such as one-way roads, two-way roads, intersections, and roundabouts, each with different distribution characteristics of salient areas.

[0003] In related technologies, traffic surveillance video eye-tracking databases play a supporting role in traffic video saliency prediction methods. However, currently widely used traffic scene eye-tracking databases, such as the "Traffic Gaze" database for first-person view driving videos and the "Personalized Situation Awareness of Drivers" (PSAD) database, are all designed for first-person view videos. There is currently no fixed-viewpoint traffic surveillance video eye-tracking database, and this scarcity of databases hinders the development of traffic surveillance video saliency prediction models. Furthermore, existing traffic video saliency prediction methods are not optimized for multi-scenario traffic surveillance, resulting in poor prediction accuracy. Therefore, improving the saliency prediction accuracy of traffic surveillance videos is a pressing issue that needs to be addressed. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a method for generating a traffic video saliency prediction model and a method for predicting traffic video saliency.

[0005] Firstly, this disclosure provides a method for generating a traffic video saliency prediction model, including:

[0006] Obtain a traffic video training sample set; the traffic video training sample set includes: multiple traffic video segments and a saliency heatmap for each video frame; wherein, each traffic video segment includes multiple consecutive video frames;

[0007] An initial video saliency prediction model is constructed; the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork;

[0008] Based on the traffic video training sample set, the prior knowledge learning subnetwork is pre-trained to learn prior knowledge of traffic videos and obtain a preset video saliency prediction model.

[0009] Based on the learned prior knowledge of traffic videos, the video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned and trained to obtain the target video saliency prediction model.

[0010] As an optional implementation of this disclosure, obtaining the traffic video training sample set includes:

[0011] Acquire multiple traffic video clips from various traffic scenarios;

[0012] Collect attention data from multiple observers in the multiple traffic video clips, and obtain multiple attention data for each video frame;

[0013] Based on the multiple points of interest data for each video frame, a saliency heatmap for each video frame is generated.

[0014] Based on the multiple traffic video clips and the saliency heatmap of each video frame, a traffic monitoring video sample set is constructed.

[0015] As an optional implementation of this disclosure, the step of pre-training the prior knowledge learning subnetwork based on the traffic video training sample set to learn prior knowledge of traffic videos and obtain a preset video saliency prediction model includes:

[0016] The multiple traffic video clips are input into the prior knowledge learning subnetwork to learn prior knowledge about traffic videos and obtain the predicted salient object map of the last video frame of each video clip.

[0017] A first loss function is determined based on the actual salient object map and the predicted salient object map for each video segment;

[0018] The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain the preset video saliency prediction model.

[0019] As an optional implementation of this disclosure, the step of fine-tuning the video saliency prediction subnetwork in the preset video saliency prediction model based on learned prior knowledge of traffic videos to obtain the target video saliency prediction model includes:

[0020] The multiple traffic video clips are input into the preset video saliency prediction model to obtain a predicted saliency heatmap for each video frame of each video clip.

[0021] The second loss function is determined based on the predicted salient heatmap of each video frame and the actual salient heatmap of each video frame;

[0022] The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain the target video saliency prediction model.

[0023] As an optional implementation of this disclosure, after obtaining the traffic video training sample set, the method further includes:

[0024] For each traffic video segment, target object detection is performed on each video frame based on the target detection model to obtain the target object detection results for each video frame.

[0025] Based on the target object detection results of each video frame, a car distribution map is generated for each video frame;

[0026] Calculate the similarity between the car distribution map and the saliency heatmap of each video frame to determine the degree of correlation between video saliency and vehicle distribution.

[0027] Secondly, embodiments of this disclosure provide a traffic video saliency prediction method, the method comprising:

[0028] Acquire traffic video to be predicted; the traffic video to be predicted includes multiple consecutive video frames.

[0029] The traffic video to be predicted is input into the target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on any of the traffic video saliency prediction model generation methods described above.

[0030] Thirdly, embodiments of this disclosure provide a traffic video saliency prediction model generation apparatus, the traffic video saliency prediction model generation apparatus comprising:

[0031] The sample set acquisition module is used to acquire a traffic video training sample set; the traffic video training sample set includes: multiple traffic video segments and a saliency heatmap of each video frame; wherein, each traffic video segment includes multiple consecutive video frames;

[0032] The model building module is used to build an initial video saliency prediction model; the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork;

[0033] The pre-training module is used to pre-train the prior knowledge learning sub-network based on the traffic video training sample set, learn prior knowledge of traffic videos, and obtain a preset video saliency prediction model.

[0034] The generation module is used to fine-tune the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned prior knowledge of traffic videos, so as to obtain the target video saliency prediction model.

[0035] As an optional implementation of this disclosure, the sample set acquisition module is specifically used for:

[0036] Acquire multiple traffic video clips from various traffic scenarios;

[0037] Collect attention data from multiple observers in the multiple traffic video clips, and obtain multiple attention data for each video frame;

[0038] Based on the multiple points of interest data for each video frame, a saliency heatmap for each video frame is generated.

[0039] Based on the multiple traffic video clips and the saliency heatmap of each video frame, a traffic monitoring video sample set is constructed.

[0040] As an optional implementation of this disclosure, the pre-training module is specifically used for:

[0041] The multiple traffic video clips are input into the prior knowledge learning subnetwork to learn prior knowledge about traffic videos and obtain the predicted salient object map of the last video frame of each video clip.

[0042] A first loss function is determined based on the actual salient object map and the predicted salient object map for each video segment;

[0043] The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain the preset video saliency prediction model.

[0044] As an optional implementation of this disclosure, the generation module is specifically used for:

[0045] The multiple traffic video clips are input into the preset video saliency prediction model to obtain a predicted saliency heatmap for each video frame of each video clip.

[0046] The second loss function is determined based on the predicted salient heatmap of each video frame and the actual salient heatmap of each video frame;

[0047] The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain the target video saliency prediction model.

[0048] Fourthly, embodiments of this disclosure provide a traffic video saliency prediction device, the traffic video saliency prediction device comprising:

[0049] The video acquisition module is used to acquire traffic video to be predicted; the traffic video to be predicted includes multiple consecutive video frames.

[0050] The saliency prediction module is used to input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described above.

[0051] Fifthly, embodiments of this disclosure provide an electronic device, including: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the traffic video saliency prediction model generation method described in the first aspect or any embodiment of the first aspect, as well as the traffic video saliency prediction method described in the second aspect.

[0052] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the traffic video saliency prediction model generation method as described in any embodiment of the first aspect and the traffic video saliency prediction method as described in the second aspect.

[0053] The technical solution provided in this disclosure has the following advantages compared with the prior art: A traffic video training sample set is obtained, comprising multiple traffic video segments and a saliency heatmap for each video frame. Each traffic video segment includes multiple consecutive video frames. An initial video saliency prediction model is constructed. A prior knowledge learning sub-network is pre-trained based on the traffic video training sample set to learn prior knowledge of traffic videos, resulting in a preset video saliency prediction model. Based on the learned prior knowledge of traffic videos, the video saliency prediction sub-network in the preset video saliency prediction model is fine-tuned to obtain a target video saliency prediction model. Since the traffic video training sample set includes multiple traffic video segments and a saliency heatmap for each video frame, and each traffic video segment includes multiple consecutive video frames, meaning that the traffic video training sample set covers common traffic scenarios, training with this training sample set can yield a more accurate target video saliency prediction model. Furthermore, since the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork, in the first task stage, it can learn the distribution of salient objects through the traffic video training sample set and the prior knowledge learning subnetwork. Then, in the second task stage, the video saliency prediction subnetwork is fine-tuned and trained based on the prior knowledge of traffic videos learned in the first task stage. The resulting target video saliency prediction model has higher saliency prediction accuracy, thereby improving the saliency prediction accuracy of the model in traffic monitoring video scenarios. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0055] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a traffic video saliency prediction model generation method provided in an embodiment of this disclosure;

[0057] Figure 2 This is a schematic diagram of a scenario data acquisition process provided in an embodiment of this disclosure;

[0058] Figure 3a This is a schematic diagram of the structure of a traffic video saliency prediction model provided in an embodiment of this disclosure;

[0059] Figure 3bThis is a comparative diagram of the predicted saliency heatmap of a traffic video saliency prediction method provided in this embodiment and other saliency prediction methods in the prior art;

[0060] Figure 4 This is a schematic flowchart of a traffic video saliency prediction method provided in an embodiment of this disclosure;

[0061] Figure 5 This is a schematic diagram of the structure of a traffic video saliency prediction model generation device provided in an embodiment of this disclosure;

[0062] Figure 6 This is a schematic diagram of the structure of a traffic video saliency prediction device provided in an embodiment of this disclosure;

[0063] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0064] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0065] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0066] The relational terms such as “first” and “second” used in this disclosure are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0067] In this disclosure, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments in this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0068] In some embodiments, such as Figure 1 As shown, a method for generating a traffic video saliency prediction model is provided, including the following steps S11-S14:

[0069] S11. Obtain a training sample set of traffic videos.

[0070] The traffic video training sample set includes: multiple traffic video segments and a saliency heatmap for each video frame; wherein each traffic video segment includes multiple consecutive video frames.

[0071] Specifically, multiple high-quality traffic videos collected from multiple data sources are acquired, the videos are edited into multiple video clips, and then the attention data of multiple observers are recorded using an eye tracker. The attention data of each video frame is heat-processed to obtain a saliency heatmap of each video frame, which constitutes the video training samples.

[0072] For example, collect 1000 high-quality traffic videos from multiple publicly available video websites. The videos should have a resolution of 1920x1080 and a frame rate greater than 30fps. The videos can then be edited into clips ranging from 5 to 20 seconds.

[0073] In some embodiments, step S11 (obtaining a traffic video training sample set) can be implemented in the following manner:

[0074] a. Acquire multiple traffic video clips under various traffic scenarios.

[0075] Specifically, to enhance the diversity of video scenes, the videos include eight common traffic scenarios, including two-way roads, one-way roads, intersections, highways, ramps, roundabouts, accidents, and others (complex scenarios such as overpasses and tunnels).

[0076] For example, the "Traffic Surveillance Videos 1K" database (TSV1K) contains 1,000 fixed-viewpoint traffic surveillance videos and 14,183,070 collected gaze points from 30 volunteer observers. (See also...) Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the scenario of database acquisition. Among them, Figure 2 (a) is a diagram illustrating the number of videos in different scenes in the database. Figure 2 (b) is a diagram illustrating the number of videos under different weather conditions in the database. Figure 2 (c) is a diagram illustrating the number of videos under different lighting conditions in the database. Figure 2 (d) is a schematic diagram of video footage of different traffic scenarios.

[0077] b. Collect attention data from multiple observers in the multiple traffic video clips, and obtain multiple attention data for each video frame.

[0078] Specifically, the system collects attention data from multiple observers across multiple traffic video clips, obtaining attention data from multiple observers on each video frame.

[0079] For example, an eye tracker capable of recording focus data at a frequency of 500Hz was used in the experiment. The average accuracy of the eye tracker's calibration area was 0.25-0.5 degrees. During the experiment, the video was displayed on a 21.5-inch TFT monitor with a resolution of 1920×1080, and the video was played randomly through the eye tracker's integrated software. To ensure the quality of the eye tracker's recorded data, all observers had normal or corrected vision, and each observer underwent eye tracker calibration before the experiment. Observers sat in an adjustable chair, approximately 55 cm away from the screen. Before the experiment, each observer underwent 9-point calibration and 12-point verification of the eye tracker. After the experiment began, the eye tracker recorded the observers' focus data. To alleviate observer fatigue during the experiment, the 1000 videos were divided into 4 groups, with observers required to rest for 15 minutes after each group. The total experiment time for each observer was approximately 4 hours. By randomly playing 1,000 videos, 14,183,070 points of interest were collected from an experiment with 30 observers.

[0080] c. Generate a saliency heatmap for each video frame based on the multiple points of interest data for each video frame.

[0081] Specifically, the data from multiple points of interest in each video frame are collected and used to generate a saliency heatmap frame by frame. A saliency heatmap is a tool used to visualize salient regions in an image or video. It uses color variations to represent the saliency or attention level of different regions, typically using the intensity of the color to indicate the level of saliency. The "hotter" the color (e.g., red, yellow), the higher the saliency; the "cooler" the color (e.g., blue, green), the lower the saliency.

[0082] For example, assuming there are 30 observers, 30 points of interest data can be collected for each video frame, and a salience heatmap for each video frame can be generated based on the collected 30 points of interest data.

[0083] d. Based on the multiple traffic video clips and the saliency heatmap of each video frame, construct a traffic monitoring video sample set.

[0084] Specifically, a traffic monitoring video sample set is constructed based on multiple traffic video clips and saliency heatmaps of each video frame.

[0085] By establishing a large-scale, high-quality eye-tracking database of traffic surveillance videos, covering common traffic scenarios, and using it to train a traffic video saliency prediction model, the model's saliency prediction accuracy in traffic surveillance video scenarios can be improved.

[0086] In some embodiments, after performing step S11 (obtaining the traffic video training sample set) as described above, the following steps may also be performed:

[0087] For each traffic video segment, target object detection is performed on each video frame based on the target detection model to obtain the target object detection results for each video frame.

[0088] Based on the target object detection results of each video frame, a car distribution map is generated for each video frame;

[0089] Calculate the similarity between the car distribution map and the saliency heatmap of each video frame to determine the degree of correlation between video saliency and vehicle distribution.

[0090] The object detection model can be a YOLOv8 model. YOLOv8 supports various tasks and modes, including object detection, instance segmentation, pose estimation, tracking, and classification. It provides a variety of pre-trained models to meet the needs of different scenarios.

[0091] Specifically, for each traffic video segment, target object detection is performed on each video frame using an object detection model, obtaining the target object detection results for each video frame. Based on the target object detection results for each video frame, a vehicle distribution map is generated for each video frame. The similarity between the vehicle distribution map and the saliency heatmap of each video frame is calculated to determine the degree of correlation between video saliency and vehicle distribution. By analyzing the degree of correlation between video saliency and vehicle distribution, the relevant knowledge about vehicle distribution that the initial video saliency prediction model needs to learn can be determined.

[0092] For example, firstly, object detection is performed on each frame of the video using YOLOv8. From the detection results, eight target categories strongly related to traffic (such as passenger cars, trucks, automobiles, buses, pedestrians, etc.) are selected to generate a car distribution map. Then, the similarity between the car distribution map and the saliency heatmap of each video frame is calculated. The formula for calculating the similarity is as follows:

[0093]

[0094] Wherein, Dice represents the similarity between the car distribution map and the saliency heatmap of each video frame, A represents the set of pixels in the binarized image of the car distribution map of a certain frame, B represents the set of pixels in the binarized image of the corresponding saliency heatmap of that frame, |•| represents taking all pixels as a set, and ∩ represents taking the intersection. The Dice coefficient is used to measure the similarity between two sets, with a value ranging from 0 to 1, where 1 represents complete similarity and 0 represents no difference. In this embodiment of the disclosure, the similarity between the car distribution map and the saliency heatmap is calculated to be 0.669, that is, by analyzing the above-mentioned TSV1K database, a strong correlation is found between the saliency of the video and the vehicle distribution.

[0095] S12. Construct an initial video saliency prediction model.

[0096] The initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork.

[0097] Specifically, based on the above analysis, it was found that there is a strong correlation between video saliency and vehicle distribution. Therefore, the initial video saliency prediction model is constructed by a shared encoder prior knowledge learning subnetwork and a video saliency prediction subnetwork.

[0098] For example, this disclosure provides a multi-task saliency prediction model, "Multi-task traffic surveillance saliency prediction network" (MTTS-Net), with the following model structure: Figure 3a As shown, the model consists of three parts: a backbone network, a Traffic Saliency Object Detection (TSOD) predictor, and a Video Saliency Prediction (VSP) predictor. The model first learns the distribution of salient objects through the TSOD predictor, and then continues to complete the VSP task based on the prior knowledge learned from the TSOD task.

[0099] Reference Figure 3a As shown, the backbone network is an encoder based on TMFI (Transformer-based multi-scale feature integration network), which consists of a block segmentation layer and four "Video Swin Transformer" (VST) blocks. The block segmentation layer divides the input T-frame images into non-overlapping 3D blocks, which are then sequentially fed into the four VST blocks to obtain four layers of features from low to high. Top-level features Rich in semantic information, these features are fed into a 2-dimensional pyramid pooling module (2D PPM) and a 3-dimensional pyramid pooling module (3D PPM) respectively to extract the top-level features of the backbone network output. semantic information, Features output by the "2D pyramid pooling module", This refers to the features output by the "3D Pyramid Pooling Module". The structure of 3DPPM is similar to 2D PPM, but compared to 2D PPM, 3D PPM can extract temporal correlation information from multiple consecutive video frames. For example... Figure 3a As shown in the upper right corner, the original features are subjected to 3D pooling at three different scales, then upsampled by 3D convolution, and finally concatenated with the original features before being subjected to 3D convolution to obtain the output.

[0100] Since the preceding data analysis shows a close correlation between salient regions and traffic scenarios, a Traffic Salient Object Detection (TSOD) predictor is designed to assist the backbone network in learning prior information about the traffic scenario. Feature processing is performed using the Feature Aggregation Module (FAM) proposed in the PoolNet-based saliency prediction network, such as... Figure 3a As shown, It is fed into FAM, and then fused together with it through the fusion module. , Add them together, and repeat the above operation until a salient object image is output. .

[0101] like Figure 3a As shown, a multi-level, efficient VSP predictor was designed. The features output from the four VSP blocks are first subjected to 3D convolution, and then fused from top to bottom. The fusion process is as follows:

[0102]

[0103] in, This represents the four layers of features output from the four VST blocks, from low to high. This represents the characteristics output by the 3D PPM module. Indicates that the VSP predictor will and Features after fusion This represents a 3D convolutional layer. express Upsampling layer of times, This indicates summation of elements.

[0104] Furthermore, a "Multiple alternately stacked 3D Convolutional layers and Upsampling layers" (MCU) was designed to extract features layer by layer. The MCU structure is as follows: Figure 3a As shown, the calculation method is as follows:

[0105]

[0106] in, Indicates the first The output of the layer MCU, Indicates that the VSP predictor will and Features after fusion This represents the Sigmoid activation function. This represents the ReLU activation function.

[0107] Finally, the MCU output The images are stitched together and then passed through a 3D convolutional layer to obtain a saliency map. .

[0108] S13. Based on the traffic video training sample set, the prior knowledge learning sub-network is pre-trained to learn prior knowledge of traffic videos and obtain a preset video saliency prediction model.

[0109] Considering the different design objectives of the TSOD and VSP predictors, a two-stage training strategy is adopted during training.

[0110] In some embodiments, step S13 (pre-training the prior knowledge learning sub-network based on the traffic video training sample set to learn prior knowledge of traffic videos and obtain a preset video saliency prediction model) can be implemented in the following way:

[0111] 1) Input the multiple traffic video clips into the prior knowledge learning subnetwork to learn prior knowledge of traffic videos and obtain the predicted salient object map of the last video frame of each video clip.

[0112] Specifically, multiple traffic video clips are input into the prior knowledge learning subnetwork to learn prior knowledge about traffic videos and obtain the predicted salient object map of the last video frame of each video clip.

[0113] For example, in the TSOD task, it should be noted that the salient object map shows the outlines of each salient object.

[0114] 2) Determine the first loss function based on the actual salient object map and the predicted salient object map for each video segment.

[0115] 3) Adjust the weight parameters of the initial video saliency prediction model according to the first loss function to obtain the preset video saliency prediction model.

[0116] Specifically, in order to fully learn prior traffic knowledge, only the TSOD predictor is trained in the first stage, and the first loss function uses the binary cross-entropy loss (BCE Loss), calculated as follows:

[0117]

[0118] in, The loss function represents the difference between the actual salient object map and the predicted salient object map for each video segment. Represents a diagram of actual salient objects. This represents a map predicting salient objects. Indicates the total number of pixels in the frame. express In the The value of each pixel. express In the The value of each pixel. It should be noted that the Mask R-CNN network can be used to obtain the segmentation mask of the vehicle as the ground truth for salient objects. .

[0119] S14. Based on the learned prior knowledge of traffic videos, fine-tune the video saliency prediction sub-network in the preset video saliency prediction model to obtain the target video saliency prediction model.

[0120] In some embodiments, step S14 (fine-tuning the video saliency prediction sub-network in the preset video saliency prediction model based on the learned prior knowledge of traffic videos to obtain the target video saliency prediction model) can be implemented in the following way:

[0121] (1) Input the multiple traffic video segments into the preset video saliency prediction model to obtain the predicted saliency heatmap of each video frame of each video segment.

[0122] (2) Determine the second loss function based on the predicted salient heatmap of each video frame and the actual salient heatmap of each video frame.

[0123] (3) Adjust the weight parameters of the preset video saliency prediction model according to the second loss function to obtain the target video saliency prediction model.

[0124] Specifically, in the second phase of training, the backbone network is initialized with the parameters trained in the first phase, and then a combined loss function is used. To train the backbone network and VSP predictor. Loss function based on linear correlation coefficient and KL divergence loss function constitute, The calculation formula is as follows:

[0125]

[0126] in, Heatmap showing predictive significance And actual significance heatmap The linear correlation coefficient loss function, Heatmap showing predictive significance And actual significance heatmap covariance, This represents the function for calculating standard deviation.

[0127] The calculation formula is as follows:

[0128]

[0129] in, Heatmap showing predictive significance And actual significance heatmap The KL divergence loss function, This represents a pixel in the actual saliency heatmap. The probability of belonging to a significant region. This represents a pixel in the predictive significance heatmap. The probability of belonging to a significant region.

[0130] Therefore, the second loss function is defined as follows:

[0131]

[0132] in, Heatmap showing predictive significance And actual significance heatmap The loss function, i.e., the second loss function, Heatmap showing predictive significance And actual significance heatmap The KL divergence loss function, Heatmap showing predictive significance And actual significance heatmap The linear correlation coefficient loss function.

[0133] In this embodiment of the disclosure, reference is made to Figure 3b As shown, Figure 3b This diagram illustrates a comparison of the predicted saliency heatmaps of a traffic video saliency prediction method provided in this disclosure with other existing saliency prediction methods. To ensure fairness in the comparison, all methods, including TMFI, STSANet (Spatio-temporal self-attention network), ViNet (Visual Saliency Network), and TASED (Temporally-Aggregating SpatialEncoder-Decoder Network), were retrained using the TSV1K training set. The performance of this method is then compared with other existing saliency prediction methods. Figure 3b Column (1) shows images of several video frames from the original input video; column (2) shows the actual saliency heatmaps generated for these video frames based on multiple collected points of interest data; column (3) shows the corresponding predicted saliency heatmaps obtained using this method for these video frames; column (4) shows the corresponding predicted saliency heatmaps obtained using the TMFI network for these video frames; column (5) shows the corresponding predicted saliency heatmaps obtained using the STSANet network for these video frames; column (6) shows the corresponding predicted saliency heatmaps obtained using the ViNet network for these video frames; and column (7) shows the corresponding predicted saliency heatmaps obtained using the TASED network for these video frames. It can be observed that, compared with other saliency models, the saliency maps predicted by this method are closer to reality. These results demonstrate the effectiveness and generalization ability of this method in traffic video saliency prediction.

[0134] The traffic video saliency prediction model generation method disclosed herein obtains a traffic video training sample set, which includes multiple traffic video segments and saliency heatmaps for each video frame. Each traffic video segment includes multiple consecutive video frames. An initial video saliency prediction model is constructed. A prior knowledge learning subnetwork is pre-trained based on the traffic video training sample set to learn prior knowledge of traffic videos, thus obtaining a preset video saliency prediction model. Based on the learned prior knowledge of traffic videos, the video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned to obtain a target video saliency prediction model. Since the traffic video training sample set includes multiple traffic video segments and saliency heatmaps for each video frame, and each traffic video segment includes multiple consecutive video frames, meaning that the traffic video training sample set covers common traffic scenarios, training with this training sample set can yield a more accurate target video saliency prediction model. Furthermore, since the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork, in the first task stage, it can learn the distribution of salient objects through the traffic video training sample set and the prior knowledge learning subnetwork. Then, in the second task stage, the video saliency prediction subnetwork is fine-tuned and trained based on the prior knowledge of traffic videos learned in the first task stage. The resulting target video saliency prediction model has higher saliency prediction accuracy, thereby improving the saliency prediction accuracy of the model in traffic monitoring video scenarios.

[0135] Figure 4 This is a flowchart illustrating a traffic video saliency prediction method provided in an embodiment of this disclosure, as shown below. Figure 4 As shown in the embodiments of this disclosure, a traffic video saliency prediction method includes:

[0136] S41. Obtain traffic video to be predicted.

[0137] The traffic video to be predicted includes multiple consecutive video frames.

[0138] Specifically, it involves acquiring traffic videos to be predicted, including multiple consecutive video frames, within a traffic scenario.

[0139] For example, acquiring traffic video of a highway intersection to be predicted, or acquiring traffic video of a ramp to be predicted, or acquiring traffic video of a roundabout to be predicted, etc.

[0140] S42. Input the traffic video to be predicted into the target video saliency prediction model to obtain the saliency heatmap corresponding to each video frame of the traffic video to be predicted.

[0141] The target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described above.

[0142] Specifically, the traffic video to be predicted is input into the target video saliency prediction model to obtain the saliency heatmap corresponding to each video frame of the traffic video to be predicted.

[0143] For example, a traffic video containing 32 video frames can be input into a target video saliency prediction model to obtain a saliency heatmap corresponding to each of the 32 video frames.

[0144] The traffic video saliency prediction method disclosed herein obtains a traffic video to be predicted, wherein the traffic video to be predicted comprises multiple consecutive video frames; the traffic video to be predicted is input into a target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted. Since the saliency prediction accuracy of the target video saliency prediction model generated by the above-described traffic video saliency prediction model generation method is improved, using the target video saliency prediction model to perform saliency prediction on the traffic video to be predicted can improve the saliency prediction accuracy of the video to be predicted.

[0145] In some embodiments, refer to Figure 5 As shown, a traffic video saliency prediction model generation device 500 is provided, comprising:

[0146] The sample set acquisition module 510 is used to acquire a traffic video training sample set; the traffic video training sample set includes: multiple traffic video segments and a saliency heatmap of each video frame; wherein, each traffic video segment includes multiple consecutive video frames;

[0147] The model building module 520 is used to build an initial video saliency prediction model; the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork;

[0148] The pre-training module 530 is used to pre-train the prior knowledge learning sub-network based on the traffic video training sample set, learn prior knowledge of traffic videos, and obtain a preset video saliency prediction model.

[0149] The generation module 540 is used to fine-tune the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned prior knowledge of traffic videos, so as to obtain the target video saliency prediction model.

[0150] As an optional implementation of this disclosure, the sample set acquisition module 510 is specifically used for:

[0151] Acquire multiple traffic video clips from various traffic scenarios;

[0152] Collect attention data from multiple observers in the multiple traffic video clips, and obtain multiple attention data for each video frame;

[0153] Based on the multiple points of interest data for each video frame, a saliency heatmap for each video frame is generated.

[0154] Based on the multiple traffic video clips and the saliency heatmap of each video frame, a traffic monitoring video sample set is constructed.

[0155] As an optional implementation of this disclosure, the pre-training module is specifically used for:

[0156] The multiple traffic video clips are input into the prior knowledge learning subnetwork to learn prior knowledge about traffic videos and obtain the predicted salient object map of the last video frame of each video clip.

[0157] A first loss function is determined based on the actual salient object map and the predicted salient object map for each video segment;

[0158] The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain the preset video saliency prediction model.

[0159] As an optional implementation of this disclosure, the generation module is specifically used for:

[0160] The multiple traffic video clips are input into the preset video saliency prediction model to obtain a predicted saliency heatmap for each video frame of each video clip.

[0161] The second loss function is determined based on the predicted salient heatmap of each video frame and the actual salient heatmap of each video frame;

[0162] The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain the target video saliency prediction model.

[0163] The traffic video saliency prediction model generation device disclosed herein acquires a traffic video training sample set, which includes multiple traffic video segments and saliency heatmaps for each video frame. Each traffic video segment includes multiple consecutive video frames. An initial video saliency prediction model is constructed. Based on the traffic video training sample set, a prior knowledge learning sub-network is pre-trained to learn prior knowledge about traffic videos, resulting in a preset video saliency prediction model. Based on the learned prior knowledge about traffic videos, the video saliency prediction sub-network in the preset video saliency prediction model is fine-tuned to obtain a target video saliency prediction model. Since the traffic video training sample set includes multiple traffic video segments and saliency heatmaps for each video frame, and each traffic video segment includes multiple consecutive video frames, meaning that the traffic video training sample set covers common traffic scenarios, training with this training sample set can yield a more accurate target video saliency prediction model. Furthermore, since the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork, in the first task stage, it can learn the distribution of salient objects through the traffic video training sample set and the prior knowledge learning subnetwork. Then, in the second task stage, the video saliency prediction subnetwork is fine-tuned and trained based on the prior knowledge of traffic videos learned in the first task stage. The resulting target video saliency prediction model has higher saliency prediction accuracy, thereby improving the saliency prediction accuracy of the model in traffic monitoring video scenarios.

[0164] This disclosure provides a traffic video saliency prediction device 600, which includes:

[0165] Video acquisition module 610 is used to acquire traffic video to be predicted; the traffic video to be predicted includes multiple consecutive video frames;

[0166] The saliency prediction module 620 is used to input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described above.

[0167] The traffic video saliency prediction method disclosed herein obtains a traffic video to be predicted, wherein the traffic video to be predicted comprises multiple consecutive video frames; the traffic video to be predicted is input into a target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted. Since the saliency prediction accuracy of the target video saliency prediction model generated by the above-described traffic video saliency prediction model generation method is improved, using the target video saliency prediction model to perform saliency prediction on the traffic video to be predicted can improve the saliency prediction accuracy of the video to be predicted.

[0168] Specific limitations regarding the traffic video saliency prediction model generation device and the traffic video saliency prediction device can be found in the limitations regarding the traffic video saliency prediction model generation method and the traffic video saliency prediction method described above, and will not be repeated here. Each module in the aforementioned traffic video saliency prediction model generation device and traffic video saliency prediction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of the processor, or stored in the processor of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0169] This disclosure also provides an electronic device. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 7 As shown, the electronic device provided in this embodiment includes a memory 71 and a processor 72. The memory 71 stores computer programs; the processor 72 executes the steps of any embodiment of the fault identification method for the image acquisition device provided in the above method embodiments when the computer program is invoked. The electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the electronic device provides computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the computer program is executed by the processor, it implements a fault identification method for an image acquisition device. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad provided on the casing of a computer device, or an external keyboard, touchpad, or mouse, etc.

[0170] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. Specific electronic devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] In some embodiments, the traffic video saliency prediction model generation apparatus provided herein can be implemented as a computer, and the computer program can be implemented in, for example, Figure 7 The computer program, comprised of various program modules, causes the processor to execute the steps in the fault identification method for the image acquisition device of the electronic device described in the various embodiments of this disclosure.

[0172] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the fault identification method for the image acquisition device provided in the above-described method embodiments.

[0173] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0174] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0175] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0176] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0177] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0178] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a traffic video saliency prediction model, characterized in that, The method includes: Obtain a traffic video training sample set; the traffic video training sample set includes: multiple traffic video segments and a saliency heatmap for each video frame; wherein, each traffic video segment includes multiple consecutive video frames; An initial video saliency prediction model is constructed; the initial video saliency prediction model consists of a prior knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork; the structure of the initial video saliency prediction model includes: a backbone network, a traffic salient object detection task predictor, and a video saliency prediction predictor; The backbone network consists of a block segmentation layer and four video variable window transformer blocks; The traffic salient object detection task predictor includes a feature enhancement module and a fusion module proposed in the PoolNet salient prediction network; The video saliency prediction predictor includes 3D convolution, a 3D pyramid pooling module, multi-layer convolution, and upsampling; Based on the traffic video training sample set, the prior knowledge learning subnetwork is pre-trained to learn prior knowledge of traffic videos and obtain a preset video saliency prediction model. Based on the learned prior knowledge of traffic videos, the video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned and trained to obtain the target video saliency prediction model.

2. The method according to claim 1, characterized in that, The acquisition of the traffic video training sample set includes: Acquire multiple traffic video clips from various traffic scenarios; Collect attention data from multiple observers in the multiple traffic video clips, and obtain multiple attention data for each video frame; Based on the multiple points of interest data for each video frame, a saliency heatmap for each video frame is generated. Based on the multiple traffic video clips and the saliency heatmap of each video frame, a traffic monitoring video sample set is constructed.

3. The method according to claim 1, characterized in that, The process of pre-training the prior knowledge learning sub-network based on the traffic video training sample set to learn prior knowledge of traffic videos and obtain a preset video saliency prediction model includes: The multiple traffic video clips are input into the prior knowledge learning subnetwork to learn prior knowledge about traffic videos and obtain the predicted salient object map of the last video frame of each video clip. A first loss function is determined based on the actual salient object map and the predicted salient object map for each video segment; The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain the preset video saliency prediction model.

4. The method according to claim 1, characterized in that, The method involves fine-tuning and training the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned prior knowledge of traffic videos to obtain the target video saliency prediction model, including: The multiple traffic video clips are input into the preset video saliency prediction model to obtain a predicted saliency heatmap for each video frame of each video clip. The second loss function is determined based on the predicted salient heatmap of each video frame and the actual salient heatmap of each video frame; The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain the target video saliency prediction model.

5. The method according to claim 1, characterized in that, After obtaining the traffic video training sample set, the process also includes: For each traffic video segment, target object detection is performed on each video frame based on the target detection model to obtain the target object detection results for each video frame. Based on the target object detection results of each video frame, a car distribution map is generated for each video frame; Calculate the similarity between the car distribution map and the saliency heatmap of each video frame to determine the degree of correlation between video saliency and vehicle distribution.

6. A traffic video saliency prediction method, characterized in that, The method includes: Acquire traffic video to be predicted; the traffic video to be predicted includes multiple consecutive video frames. The traffic video to be predicted is input into the target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method according to any one of claims 1-5.

7. A traffic video saliency prediction model generation device, characterized in that, The traffic video saliency prediction model generation device includes: The sample set acquisition module is used to acquire a traffic video training sample set; the traffic video training sample set includes: multiple traffic video segments and a saliency heatmap of each video frame; wherein, each traffic video segment includes multiple consecutive video frames; The model building module is used to build an initial video saliency prediction model; the initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork; the structure of the initial video saliency prediction model includes: a backbone network, a traffic salient object detection task predictor, and a video saliency prediction predictor; The backbone network consists of a block segmentation layer and four video variable window transformer blocks; The traffic salient object detection task predictor includes a feature enhancement module and a fusion module proposed in the PoolNet salient prediction network; The video saliency prediction predictor includes 3D convolution, a 3D pyramid pooling module, multi-layer convolution, and upsampling; The pre-training module is used to pre-train the prior knowledge learning sub-network based on the traffic video training sample set, learn prior knowledge of traffic videos, and obtain a preset video saliency prediction model. The generation module is used to fine-tune the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned prior knowledge of traffic videos, so as to obtain the target video saliency prediction model.

8. A traffic video saliency prediction device, characterized in that, The traffic video saliency prediction device includes: The video acquisition module is used to acquire traffic video to be predicted; the traffic video to be predicted includes multiple consecutive video frames. A saliency prediction module is used to input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heatmap corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method according to any one of claims 1-6.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the traffic video saliency prediction model generation method according to any one of claims 1 to 5, or the traffic video saliency prediction method according to claim 6.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the traffic video saliency prediction model generation method according to any one of claims 1 to 5, or the traffic video saliency prediction method according to claim 6.

Images