Traffic video saliency prediction model generation method and traffic video saliency prediction method

Abstract

The invention relates to a traffic video saliency prediction model generation method and a traffic video saliency prediction method. The traffic video saliency prediction model generation method comprises the following steps: acquiring a traffic video training sample set; the traffic video training sample set comprises a plurality of traffic video clips and a saliency thermodynamic diagram of each video frame; wherein each traffic video clip comprises a plurality of continuous video frames; constructing an initial video saliency prediction model; the initial video saliency prediction model is composed of a priori knowledge learning sub-network and a video saliency prediction sub-network which share an encoder; based on the traffic video training sample set, pre-training the priori knowledge learning sub-network, learning traffic video priori knowledge, and obtaining a preset video saliency prediction model; and performing fine tuning training on the video saliency prediction sub-network in the preset video saliency prediction model based on the learned traffic video priori knowledge to obtain a target video saliency prediction model.

Description

Technical Field

[0001] The present disclosure relates to the field of digital image processing vision, and in particular to a traffic video saliency prediction model generation method and a traffic video saliency prediction method. Background Art

[0002] Video saliency prediction is to predict the direction of human gaze. Accurate prediction of saliency can improve the efficiency of traffic surveillance video processing, thereby improving the efficiency of discovering key information from massive videos. There are obvious differences between traffic surveillance videos and ordinary scene videos. Traffic surveillance has many scenes, such as one-way roads, two-way roads, intersections, roundabouts, etc. The salient areas in different scenes have different distribution characteristics.

[0003] In the related art, the traffic surveillance video eye tracking database plays a supporting role in the traffic video saliency prediction method of traffic surveillance video. However, the currently widely used traffic scene eye tracking databases, such as the "Traffic Gaze" database TrafficGaze for first-person driving videos and the "Personalized Situation Awareness of Drivers" database (PSAD), are all for first-person videos, and there is currently no fixed-perspective traffic surveillance video eye tracking database. The scarcity of the database hinders the development of traffic surveillance video saliency prediction models. In addition, the existing traffic video saliency prediction methods are not optimized for multi-scenario traffic monitoring, resulting in poor prediction accuracy. Therefore, how to improve the saliency prediction accuracy of traffic surveillance videos is a problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a traffic video saliency prediction model generation method and a traffic video saliency prediction method.

[0005] In a first aspect, the present disclosure provides a method for generating a traffic video saliency prediction model, comprising:

[0006] Obtaining a traffic video training sample set; the traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame; wherein each traffic video clip includes a plurality of continuous video frames;

[0007] Constructing an initial video saliency prediction model; the initial video saliency prediction model is composed of a prior knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork;

[0008] Based on the traffic video training sample set, pre-training the prior knowledge learning sub-network, learning the traffic video prior knowledge, and obtaining a preset video saliency prediction model;

[0009] Based on the learned prior knowledge of traffic videos, the video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned and trained to obtain a target video saliency prediction model.

[0010] As an optional implementation of the embodiment of the present disclosure, the step of obtaining a traffic video training sample set includes:

[0011] Obtain multiple traffic video clips in various traffic scenarios;

[0012] Collecting the focus point data of multiple observers in the multiple traffic video clips to obtain the multiple focus point data of each video frame;

[0013] Generating a saliency heat map of each video frame according to the multiple focus point data of each video frame;

[0014] A traffic monitoring video sample set is constructed based on the multiple traffic video clips and the saliency heat map of each video frame.

[0015] As an optional implementation of the embodiment of the present disclosure, the prior knowledge learning subnetwork is pre-trained based on the traffic video training sample set to learn the traffic video prior knowledge and obtain a preset video saliency prediction model, including:

[0016] Inputting the plurality of traffic video clips into the prior knowledge learning subnetwork, learning the traffic video prior knowledge, and obtaining a predicted salient object map of the last video frame of each video clip;

[0017] Determining a first loss function according to the actual salient object map of each video clip and the predicted salient object map;

[0018] The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain a preset video saliency prediction model.

[0019] As an optional implementation of the embodiment of the present disclosure, based on the learned prior knowledge of traffic videos, fine-tuning the video saliency prediction subnetwork in the preset video saliency prediction model to obtain the target video saliency prediction model includes:

[0020] Inputting the plurality of traffic video clips into the preset video saliency prediction model to obtain a predicted saliency heat map of each video frame of each video clip;

[0021] Determining a second loss function according to the predicted saliency heat map of each video frame and the actual saliency heat map of each video frame;

[0022] The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain a target video saliency prediction model.

[0023] As an optional implementation of the embodiment of the present disclosure, after obtaining the traffic video training sample set, the method further includes:

[0024] For each traffic video clip, target object detection is performed on each video frame based on the target detection model to obtain the target object detection result of each video frame;

[0025] Based on the target object detection result of each video frame, generating a car distribution map of each video frame;

[0026] The similarity between the car distribution map of each video frame and the saliency heat map of each video frame is calculated to determine the degree of correlation between video saliency and vehicle distribution.

[0027] In a second aspect, an embodiment of the present disclosure provides a method for predicting saliency of a traffic video, the method comprising:

[0028] Acquire a traffic video to be predicted; the traffic video to be predicted includes a plurality of continuous video frames;

[0029] The traffic video to be predicted is input into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method according to any one of claims 1-5.

[0030] In a third aspect, an embodiment of the present disclosure provides a traffic video saliency prediction model generation device, the traffic video saliency prediction model generation device comprising:

[0031] A sample set acquisition module is used to acquire a traffic video training sample set; the traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame; wherein each traffic video clip includes a plurality of continuous video frames;

[0032] A model building module, used to build an initial video saliency prediction model; the initial video saliency prediction model is composed of a priori knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork;

[0033] A pre-training module, used to pre-train the prior knowledge learning sub-network based on the traffic video training sample set, learn the traffic video prior knowledge, and obtain a preset video saliency prediction model;

[0034] The generation module is used to fine-tune the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned prior knowledge of traffic videos to obtain a target video saliency prediction model.

[0035] As an optional implementation of the embodiment of the present disclosure, the sample set acquisition module is specifically used to:

[0036] Obtain multiple traffic video clips in various traffic scenarios;

[0037] Collecting the focus point data of multiple observers in the multiple traffic video clips to obtain the multiple focus point data of each video frame;

[0038] Generating a saliency heat map of each video frame according to the multiple focus point data of each video frame;

[0039] A traffic monitoring video sample set is constructed based on the multiple traffic video clips and the saliency heat map of each video frame.

[0040] As an optional implementation of the embodiment of the present disclosure, the pre-training module is specifically used for:

[0041] Inputting the plurality of traffic video clips into the prior knowledge learning subnetwork, learning the traffic video prior knowledge, and obtaining a predicted salient object map of the last video frame of each video clip;

[0042] Determining a first loss function according to the actual salient object map of each video clip and the predicted salient object map;

[0043] The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain a preset video saliency prediction model.

[0044] As an optional implementation of the embodiment of the present disclosure, the generating module is specifically used for:

[0045] Inputting the plurality of traffic video clips into the preset video saliency prediction model to obtain a predicted saliency heat map of each video frame of each video clip;

[0046] Determining a second loss function according to the predicted saliency heat map of each video frame and the actual saliency heat map of each video frame;

[0047] The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain a target video saliency prediction model.

[0048] In a fourth aspect, an embodiment of the present disclosure provides a traffic video saliency prediction device, the traffic video saliency prediction device comprising:

[0049] A video acquisition module, used to acquire a traffic video to be predicted; the traffic video to be predicted includes a plurality of continuous video frames;

[0050] A saliency prediction module is used to input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described in any one of claims 1-6.

[0051] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory and a processor; the memory stores a computer program, and when the processor executes the computer program, it implements the traffic video significance prediction model generation method described in the first aspect or any embodiment of the first aspect and the traffic video significance prediction method described in the second aspect.

[0052] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating a traffic video significance prediction model as described in any embodiment of the first aspect and the method for predicting traffic video significance as described in the second aspect are implemented.

[0053] Compared with the prior art, the technical solution provided by the embodiment of the present disclosure has the following advantages: obtaining a traffic video training sample set, wherein the traffic video training sample set includes: multiple traffic video clips and a saliency heat map of each video frame, each traffic video clip includes multiple continuous video frames, constructing an initial video saliency prediction model, pre-training a priori knowledge learning subnetwork based on the traffic video training sample set, learning traffic video prior knowledge, obtaining a preset video saliency prediction model, and fine-tuning the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned traffic video prior knowledge, to obtain a target video saliency prediction model. Since the traffic video training sample set includes multiple traffic video clips and a saliency heat map of each video frame, and each traffic video clip includes multiple continuous video frames, that is, the traffic video training sample set covers common traffic scenes, so using the training sample set for training can obtain a more accurate target video saliency prediction model. Since the initial video saliency prediction model is composed of the prior knowledge learning subnetwork of the shared encoder and the video saliency prediction subnetwork, in the first task stage, the distribution of salient objects can be learned through the traffic video training sample set and the prior knowledge learning subnetwork. Then, in the second task stage, the video saliency prediction subnetwork is fine-tuned based on the traffic video prior knowledge learned in the first task stage. The obtained target video saliency prediction model has higher saliency prediction accuracy, thereby improving the saliency prediction accuracy of the model in traffic monitoring video scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0056] Figure 1 It is a flow chart of a method for generating a traffic video saliency prediction model provided by an embodiment of the present disclosure;

[0057] Figure 2 It is a schematic diagram of scene data of a database collection process provided by an embodiment of the present disclosure;

[0058] Figure 3a is a schematic diagram of a traffic video saliency prediction model structure provided by an embodiment of the present disclosure;

[0059] Figure 3bIt is a schematic diagram comparing the predicted significance heat maps of a traffic video significance prediction method provided by an embodiment of the present disclosure and other significance prediction methods in the prior art;

[0060] Figure 4 It is a flow chart of a traffic video saliency prediction method provided by an embodiment of the present disclosure;

[0061] Figure 5 It is a structural schematic diagram of a traffic video saliency prediction model generation device provided by an embodiment of the present disclosure;

[0062] Figure 6 It is a structural schematic diagram of a traffic video saliency prediction device provided by an embodiment of the present disclosure;

[0063] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0066] Relational terms such as “first” and “second” in the description and claims of the present disclosure are merely used to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0067] In the embodiments of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0068] In some embodiments, Figure 1 As shown, a method for generating a traffic video saliency prediction model is provided, comprising the following steps S11-S14:

[0069] S11. Obtain a traffic video training sample set.

[0070] The traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame; wherein each traffic video clip includes a plurality of continuous video frames.

[0071] Specifically, multiple high-quality traffic videos collected from multiple data sources are obtained, the videos are edited into multiple video clips, and then the focus data of multiple observers are recorded through an eye tracker. The focus data of each frame of video is thermally processed to obtain a saliency heat map of each video frame to constitute a video training sample.

[0072] For example, 1000 high-quality traffic videos are collected from multiple video public websites, with a video resolution of 1920x1080 and a frame rate greater than 30fps. The videos can be edited into segments ranging from 5 to 20 seconds.

[0073] In some embodiments, step S11 (obtaining a traffic video training sample set) can be implemented in the following manner:

[0074] a. Obtain multiple traffic video clips in various traffic scenarios.

[0075] Specifically, in order to improve the diversity of video scenes, the video includes 8 common traffic scenes, including two-way roads, one-way roads, intersections, highways, ramps, roundabouts, accidents and others (complex scenes such as overpasses and tunnels).

[0076] For example, the database of “1000 traffic surveillance videos 1K” (TSV1K) contains 1000 fixed-view traffic surveillance videos and 14,183,070 gaze points collected from 30 volunteer observers. Figure 2 As shown, Figure 2 This is a schematic diagram of scene data in the database collection process. Figure 2 (a) is a schematic diagram of the number of videos of different scenes in the database. Figure 2 (b) is a schematic diagram of the number of videos in the database under different weather conditions. Figure 2 (c) is a schematic diagram of the number of videos with different lighting conditions in the database. Figure 2 (d) is a schematic diagram of video images of different traffic scenes.

[0077] b. Collecting the focus point data of multiple observers in the multiple traffic video clips to obtain the multiple focus point data of each video frame.

[0078] Specifically, the focus point data of multiple observers in multiple traffic video clips are collected to obtain the focus point data of multiple observers on each video frame.

[0079] Exemplarily, an eye tracker capable of recording focus data at a frequency of 500 Hz is used during the experiment, and the average accuracy of the calibration area of ​​the eye tracker is 0.25-0.5 degrees. During the experiment, the video is presented on a 21.5-inch TFT display with a resolution of 1920×1080, and the video is played randomly through the integrated software of the eye tracker. To ensure the quality of the data recorded by the eye tracker, the vision of all observers is normal or corrected to normal, and each observer has passed the eye tracker tracking calibration before the experiment. The observer sits on an adjustable chair, about 55 cm away from the screen. Before the experiment, each observer performs a 9-point calibration and a 12-point verification of the eye tracker. After the experiment begins, the eye tracker records the observer's focus data. In order to relieve the fatigue of the observer during the experiment, the 1000 videos can be divided into 4 groups, and the observer is required to rest for 15 minutes after each group is completed. The total experiment time for each observer is about 4 hours. 1000 videos were randomly played and 14,183,070 attention point data were collected from experiments with 30 observers.

[0080] c. Generate a saliency heat map of each video frame based on the multiple focus point data of each video frame.

[0081] Specifically, the data of multiple focus points of each video frame are collected and a saliency heat map is generated frame by frame. Among them, the saliency heat map is a tool for visualizing the salient areas in an image or video. It represents the saliency or attention of different areas through color changes. The depth of the color is usually used to represent the high and low saliency. The "hotter" color (such as red, yellow, etc.) indicates a higher saliency, and the "colder" color (such as blue, green, etc.) indicates a lower saliency.

[0082] For example, assuming there are 30 observers, 30 focus point data can be collected for each video frame, and a saliency heat map of each video frame is generated based on the collected 30 focus point data.

[0083] d. Based on the multiple traffic video clips and the saliency heat map of each video frame, a traffic monitoring video sample set is constructed.

[0084] Specifically, a traffic monitoring video sample set is constructed based on multiple traffic video clips and the saliency heat map of each video frame.

[0085] By establishing a large-scale, high-quality traffic surveillance video eye tracking database covering common traffic scenarios and using it to train a traffic video saliency prediction model, the model's saliency prediction accuracy in traffic surveillance video scenarios can be improved.

[0086] In some embodiments, after executing the above step S11 (obtaining a traffic video training sample set), the following steps may also be performed:

[0087] For each traffic video clip, target object detection is performed on each video frame based on the target detection model to obtain the target object detection result of each video frame;

[0088] Based on the target object detection result of each video frame, generating a car distribution map of each video frame;

[0089] The similarity between the car distribution map of each video frame and the saliency heat map of each video frame is calculated to determine the degree of correlation between video saliency and vehicle distribution.

[0090] The target detection model may be a YOLOv8 model. YOLOv8 supports multiple tasks and modes, including object detection, instance segmentation, pose estimation, tracking, and classification. It provides a variety of pre-trained models to meet the needs of different scenarios.

[0091] Specifically, for each traffic video clip, target object detection is performed on each video frame based on the target detection model to obtain the target object detection result of each video frame. Based on the target object detection result of each video frame, a car distribution map of each video frame is generated. The similarity between the car distribution map of each video frame and the saliency heat map of each video frame is calculated to determine the degree of correlation between video saliency and vehicle distribution. Through the degree of correlation between video saliency and vehicle distribution, it can be determined that the initial video saliency prediction model needs to learn relevant knowledge about car distribution.

[0092] For example, first, use YOLOv8 to perform object detection on each frame of the video, select 8 target categories that are strongly related to traffic (such as passenger cars, trucks, cars, buses, pedestrians, etc.) from the detection results to generate a car distribution map, and then calculate the similarity between the car distribution map of each video frame and the saliency heat map of each video frame. The similarity calculation formula is as follows:

[0093]

[0094] Among them, Dice represents the similarity between the car distribution map of each video frame and the saliency heat map of each video frame, A represents the pixel set of the image after the car distribution map of a certain frame is binarized, B represents the pixel set of the image after the saliency heat map corresponding to the frame is binarized, |·| represents taking all pixels as a set, and ∩ represents taking the intersection. The Dice coefficient is used to measure the similarity of two sets, and the value range is from 0 to 1, where 1 represents exactly the same and 0 represents no difference. In the embodiment of the present disclosure, the similarity between the car distribution map and the saliency heat map is calculated to be 0.669, that is, by analyzing the above-mentioned TSV1K database, it is found that the saliency of the video is strongly correlated with the vehicle distribution.

[0095] S12. Construct an initial video saliency prediction model.

[0096] The initial video saliency prediction model is composed of a prior knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork.

[0097] Specifically, based on the above analysis, it is found that there is a strong correlation between the saliency of the video and the vehicle distribution, so the constructed initial video saliency prediction model consists of a prior knowledge learning subnetwork of the shared encoder and a video saliency prediction subnetwork.

[0098] For example, the present disclosure provides a multi-task saliency prediction model "Multi-task traffic surveillance saliency prediction network" (MTTS-Net), the model structure is as follows: Figure 3a As shown in the figure, the model consists of three parts: the backbone network, the TSOD predictor for the traffic salient object detection task, and the VSP (Video Saliency Prediction) predictor. The model first learns the distribution of salient objects through the TSOD predictor, and then continues to complete the VSP task based on the prior knowledge learned from the TSOD task.

[0099] Reference Figure 3a As shown in the figure, the backbone network is an encoder based on TMFI (Transformer-based multi-scale feature integration network), which consists of a block segmentation layer and four "Video Variable Window Transformer" (VST) blocks. The block segmentation layer divides the input T frame image into non-overlapping 3D blocks, which are then fed into four VST blocks in turn to obtain four layers of features from low to high: The top-level features Rich in semantic information, it is sent to the "2-dimension Pyramid Pooling Module, 2D PPM" and "3-dimension Pyramid Pooling Module, 3D PPM" to extract the "top-level features of the backbone network output" The semantic information of is the feature output by the “two-dimensional pyramid pooling module”. is the feature output by the "3D Pyramid Pooling Module". The structure of 3DPPM is similar to 2D PPM. Compared with 2D PPM, 3D PPM can extract the temporal correlation information of multiple consecutive video frames. Figure 3a As shown in the upper right corner, the original features are pooled at three different scales, upsampled through 3D convolution, and finally concatenated with the original features and then convolved through 3D convolution to obtain the output.

[0100] Since the data analysis in the previous article shows that salient areas are closely related to traffic scenes, a traffic salient object detection task TSOD predictor is designed to assist the backbone network in learning prior information about traffic scenes. The “Feature Aggregation Module” (FAM) proposed in the pooling-based saliency prediction network PoolNet is used for feature processing, such as Figure 3a As shown, It is then fed into FAM and then fused with Add, repeat the above operation until the output salient object map P v .

[0101] like Figure 3a As shown in Figure 2, a multi-level efficient VSP predictor is designed. The features output by the four VSP blocks are first subjected to 3D convolution and then Perform top-down fusion. The fusion process is as follows:

[0102]

[0103] in, Represents the 4-layer features from low to high output by the 4 VST blocks, Represents the characteristics of the 3D PPM module output, Indicates that the VSP predictor will and The fused features, Conv(·) represents the 3D convolution layer, Up ×i (·)(i=2,4,8) represents the i-fold upsampling layer, Represents the sum of elements.

[0104] Furthermore, a “Multiple alternately stacked 3D Convolutional layers and Upsampling layers (MCU)” is designed to extract features layer by layer. The MCU structure is as follows: Figure 3a As shown, the calculation method is as follows:

[0105]

[0106] in, represents the output of the j-th layer MCU, Indicates that the VSP predictor will and After fusion, σ(·) represents the Sigmoid activation function, and φ(·) represents the ReLU activation function.

[0107] Finally, the MCU output After splicing, the salient map P is obtained through the 3D convolution layer. s .

[0108] S13. Based on the traffic video training sample set, pre-train the prior knowledge learning sub-network to learn the traffic video prior knowledge and obtain a preset video saliency prediction model.

[0109] Considering the differences in the design purposes of TSOD and VSP predictors, a two-stage training strategy is adopted in training.

[0110] In some embodiments, step S13 (pre-training the prior knowledge learning subnetwork based on the traffic video training sample set, learning the traffic video prior knowledge, and obtaining a preset video saliency prediction model) can be implemented as follows:

[0111] 1) Inputting the multiple traffic video clips into the prior knowledge learning subnetwork, learning the traffic video prior knowledge, and obtaining the predicted salient object map of the last video frame of each video clip.

[0112] Specifically, multiple traffic video clips are input into the prior knowledge learning subnetwork to learn the traffic video prior knowledge and obtain a predicted salient object map of the last video frame of each video clip.

[0113] For example, for the TSOD task, it should be noted that the outlines of various salient objects can be seen in the salient object map.

[0114] 2) Determine a first loss function according to the actual salient object map of each video clip and the predicted salient object map.

[0115] 3) According to the first loss function, the weight parameters of the initial video saliency prediction model are adjusted to obtain a preset video saliency prediction model.

[0116] Specifically, in order to fully learn the traffic prior knowledge, only the TSOD predictor is trained in the first stage, and the first loss function uses the binary cross entropy loss (BCE Loss), which is calculated as follows:

[0117]

[0118] Among them, L TSOD (P v ,G v ) represents the loss function between the actual salient object map of each video clip and the predicted salient object map, G v represents the actual salient object map, P v represents the predicted salient object map, N represents the total number of frame pixels, and y i Represents G v The value of the i-th pixel, p i Indicates P v The value of the i-th pixel. It should be noted that the Mask R-CNN network can be used to obtain the segmentation mask of the vehicle as the true value G of the salient object. v .

[0119] S14. Based on the learned prior knowledge of traffic videos, fine-tune the video saliency prediction subnetwork in the preset video saliency prediction model to obtain a target video saliency prediction model.

[0120] In some embodiments, the above step S14 (based on the learned prior knowledge of traffic videos, fine-tuning the video saliency prediction subnetwork in the preset video saliency prediction model to obtain the target video saliency prediction model) can be implemented as follows:

[0121] (1) Inputting the plurality of traffic video clips into the preset video saliency prediction model to obtain a predicted saliency heat map of each video frame of each video clip.

[0122] (2) Determine a second loss function based on the predicted saliency heat map of each video frame and the actual saliency heat map of each video frame.

[0123] (3) Adjusting the weight parameters of the preset video saliency prediction model according to the second loss function to obtain a target video saliency prediction model.

[0124] Specifically, in the second stage of training, the backbone network is initialized with the parameters trained in the first stage, and then a combined loss function L is used.VSP To train the backbone network and VSP predictor. VSP By the linear correlation coefficient loss function L CC And KL divergence loss function L KL Composition, L CC The calculation formula is as follows:

[0125]

[0126] Among them, L CC (P s ,G s ) represents the predicted significance heat map P s and the actual significance heat map G s The linear correlation coefficient loss function, cov(P s ,G s ) represents the predicted significance heat map P s and the actual significance heat map G s The covariance of , ρ(·) represents the standard deviation calculation function.

[0127] L KL The calculation formula is as follows:

[0128]

[0129] Among them, L KL (P s ,G s ) represents the predicted significance heat map P s and the actual significance heat map G s The KL divergence loss function, G s (x) represents the probability that a pixel point x in the actual saliency heat map belongs to the salient area, P s (x) represents the probability that a pixel point x in the predicted saliency heat map belongs to the salient area.

[0130] Therefore, the second loss function is defined as follows:

[0131] L VSP (P s ,G s )=L KL (P s ,G s )+L CC (P s ,G s ).

[0132] Among them, L VSP (P s ,G s ) represents the predicted significance heat map P s and the actual significance heat map Gs The loss function, that is, the second loss function, L KL (P s ,G s ) represents the predicted significance heat map P s and the actual significance heat map G s The KL divergence loss function, L CC (P s ,G s ) represents the predicted significance heat map P s and the actual significance heat map G s The linear correlation coefficient loss function.

[0133] In the embodiments of the present disclosure, refer to Figure 3b As shown, Figure 3b A schematic diagram of a comparison of the predicted saliency heat map of a traffic video saliency prediction method provided by the embodiment of the present disclosure and other saliency prediction methods in the prior art. In order to ensure the fairness of the comparison results, all methods are retrained using the TSV1K training set, such as TMFI, STSANet (Spatio-temporal self-attention network), ViNet (Visual Saliency Network, a neural network for visual saliency detection), and TASED (Temporally-Aggregating SpatialEncoder-Decoder Network, a neural network for visual saliency detection). The performance of this method is compared with other saliency methods in the prior art, for example, Figure 3b In the figure, column (1) is the image of some frames of the original video input; column (2) is the actual saliency heat map generated for these frames according to the collected multiple focus point data; column (3) is the corresponding predicted saliency heat map obtained by this method for these frames; column (4) is the corresponding predicted saliency heat map obtained by TMFI network for these frames; column (5) is the corresponding predicted saliency heat map obtained by STSANet network for these frames; column (6) is the corresponding predicted saliency heat map obtained by ViNet network for these frames; column (7) is the corresponding predicted saliency heat map obtained by TASED network for these frames. It can be observed that the saliency map predicted by this method is closer to the facts than other saliency models. These results show the effectiveness and generalization ability of this method in traffic video saliency prediction.

[0134] The method for generating a traffic video saliency prediction model provided by the present disclosure obtains a traffic video training sample set, wherein the traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame, each traffic video clip includes a plurality of continuous video frames, an initial video saliency prediction model is constructed, a priori knowledge learning subnetwork is pre-trained based on the traffic video training sample set, traffic video prior knowledge is learned, a preset video saliency prediction model is obtained, and based on the learned traffic video prior knowledge, a video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned to obtain a target video saliency prediction model. Since the traffic video training sample set includes a plurality of traffic video clips and a saliency heat map of each video frame, and each traffic video clip includes a plurality of continuous video frames, that is, the traffic video training sample set covers common traffic scenes, a more accurate target video saliency prediction model can be obtained by training using the training sample set. Since the initial video saliency prediction model is composed of the prior knowledge learning subnetwork of the shared encoder and the video saliency prediction subnetwork, in the first task stage, the distribution of salient objects can be learned through the traffic video training sample set and the prior knowledge learning subnetwork. Then, in the second task stage, the video saliency prediction subnetwork is fine-tuned based on the traffic video prior knowledge learned in the first task stage. The obtained target video saliency prediction model has higher saliency prediction accuracy, thereby improving the saliency prediction accuracy of the model in traffic monitoring video scenarios.

[0135] Figure 4 is a flow chart of a traffic video saliency prediction method provided by an embodiment of the present disclosure, such as Figure 4 As shown, a traffic video saliency prediction method provided by an embodiment of the present disclosure includes:

[0136] S41. Obtain the traffic video to be predicted.

[0137] The traffic video to be predicted includes a plurality of continuous video frames.

[0138] Specifically, a traffic video to be predicted including a plurality of continuous video frames is obtained in a traffic scene.

[0139] Exemplarily, a traffic video to be predicted at a highway intersection is obtained, or a traffic video to be predicted at a ramp is obtained, or a traffic video to be predicted at a roundabout is obtained, and the like.

[0140] S42: input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted.

[0141] Wherein, the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described in any one of claims 1-5.

[0142] Specifically, the traffic video to be predicted is input into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted.

[0143] Exemplarily, for example, a traffic video containing 32 video frames is input into the target video saliency prediction model to obtain a saliency heat map corresponding to each of the 32 video frames.

[0144] The traffic video saliency prediction method provided by the present disclosure obtains a traffic video to be predicted; wherein the traffic video to be predicted includes a plurality of continuous video frames; the traffic video to be predicted is input into a target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted. Since the saliency prediction accuracy of the target video saliency prediction model generated by the traffic video saliency prediction model generation method is improved, the saliency prediction accuracy of the video to be predicted can be improved by using the target video saliency prediction model to perform saliency prediction on the traffic video to be predicted.

[0145] In some embodiments, reference Figure 5 As shown, a traffic video saliency prediction model generation device 500 is provided, comprising:

[0146] The sample set acquisition module 510 is used to acquire a traffic video training sample set; the traffic video training sample set includes: a plurality of traffic video segments and a saliency heat map of each video frame; wherein each traffic video segment includes a plurality of continuous video frames;

[0147] A model building module 520, for building an initial video saliency prediction model; the initial video saliency prediction model is composed of a priori knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork;

[0148] A pre-training module 530 is used to pre-train the prior knowledge learning sub-network based on the traffic video training sample set, learn the traffic video prior knowledge, and obtain a preset video saliency prediction model;

[0149] The generation module 540 is used to fine-tune the video saliency prediction sub-network in the preset video saliency prediction model based on the learned prior knowledge of traffic videos to obtain a target video saliency prediction model.

[0150] As an optional implementation of the embodiment of the present disclosure, the sample set acquisition module 510 is specifically used to:

[0151] Obtain multiple traffic video clips in various traffic scenarios;

[0152] Collecting the focus point data of multiple observers in the multiple traffic video clips to obtain the multiple focus point data of each video frame;

[0153] Generating a saliency heat map of each video frame according to the multiple focus point data of each video frame;

[0154] A traffic monitoring video sample set is constructed based on the multiple traffic video clips and the saliency heat map of each video frame.

[0155] As an optional implementation of the embodiment of the present disclosure, the pre-training module is specifically used for:

[0156] Inputting the plurality of traffic video clips into the prior knowledge learning subnetwork, learning the traffic video prior knowledge, and obtaining a predicted salient object map of the last video frame of each video clip;

[0157] Determining a first loss function according to the actual salient object map of each video clip and the predicted salient object map;

[0158] The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain a preset video saliency prediction model.

[0159] As an optional implementation of the embodiment of the present disclosure, the generating module is specifically used for:

[0160] Inputting the plurality of traffic video clips into the preset video saliency prediction model to obtain a predicted saliency heat map of each video frame of each video clip;

[0161] Determining a second loss function according to the predicted saliency heat map of each video frame and the actual saliency heat map of each video frame;

[0162] The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain a target video saliency prediction model.

[0163] The traffic video saliency prediction model generation device provided by the present disclosure obtains a traffic video training sample set, wherein the traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame, each traffic video clip includes a plurality of continuous video frames, constructs an initial video saliency prediction model, pre-trains a priori knowledge learning subnetwork based on the traffic video training sample set, learns traffic video prior knowledge, obtains a preset video saliency prediction model, and fine-tunes the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned traffic video prior knowledge, to obtain a target video saliency prediction model. Since the traffic video training sample set includes a plurality of traffic video clips and a saliency heat map of each video frame, and each traffic video clip includes a plurality of continuous video frames, that is, the traffic video training sample set covers common traffic scenes, so a more accurate target video saliency prediction model can be obtained by training with the training sample set. Since the initial video saliency prediction model is composed of the prior knowledge learning subnetwork of the shared encoder and the video saliency prediction subnetwork, in the first task stage, the distribution of salient objects can be learned through the traffic video training sample set and the prior knowledge learning subnetwork. Then, in the second task stage, the video saliency prediction subnetwork is fine-tuned based on the traffic video prior knowledge learned in the first task stage. The obtained target video saliency prediction model has higher saliency prediction accuracy, thereby improving the saliency prediction accuracy of the model in traffic monitoring video scenarios.

[0164] The embodiment of the present disclosure provides a traffic video saliency prediction device 600, the traffic video saliency prediction device 600 comprising:

[0165] The video acquisition module 610 is used to acquire a traffic video to be predicted; the traffic video to be predicted includes a plurality of continuous video frames;

[0166] The saliency prediction module 620 is used to input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described in any one of claims 1-6.

[0167] The traffic video saliency prediction method provided by the present disclosure obtains a traffic video to be predicted; wherein the traffic video to be predicted includes a plurality of continuous video frames; the traffic video to be predicted is input into a target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted. Since the saliency prediction accuracy of the target video saliency prediction model generated by the traffic video saliency prediction model generation method is improved, the saliency prediction accuracy of the video to be predicted can be improved by using the target video saliency prediction model to perform saliency prediction on the traffic video to be predicted.

[0168] The specific limitations of the traffic video significance prediction model generation device and the traffic video significance prediction device can be found in the above-mentioned limitations of the traffic video significance prediction model generation method and the traffic video significance prediction method, which will not be repeated here. The various modules in the above-mentioned traffic video significance prediction model generation device and the traffic video significance prediction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor of the electronic device in the form of hardware, or can be stored in the processor of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0169] The present disclosure also provides an electronic device, Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device provided in this embodiment includes: a memory 71 and a processor 72, the memory 71 is used to store a computer program; the processor 72 is used to execute the steps executed in any embodiment of the fault identification method of the image acquisition device provided by the above method embodiment when calling the computer program. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a method for fault identification of an image acquisition device is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0170] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] In some embodiments, the traffic video significance prediction model generation device provided by the present disclosure can be implemented in the form of a computer program. Figure 7 The computer program composed of various program modules enables the processor to execute the steps of the fault identification method of the image acquisition device of the electronic device of each embodiment of the present disclosure described in this specification.

[0172] The embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the fault identification method for the image acquisition device provided by the above method embodiment is implemented.

[0173] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0174] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0175] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0177] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0178] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a traffic video saliency prediction model, characterized in that: The method comprises: Obtaining a traffic video training sample set; the traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame; wherein each traffic video clip includes a plurality of continuous video frames; Constructing an initial video saliency prediction model; the initial video saliency prediction model is composed of a prior knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork; Based on the traffic video training sample set, pre-training the prior knowledge learning sub-network, learning the traffic video prior knowledge, and obtaining a preset video saliency prediction model; Based on the learned prior knowledge of traffic videos, the video saliency prediction subnetwork in the preset video saliency prediction model is fine-tuned and trained to obtain a target video saliency prediction model.

2. The method according to claim 1, characterized in that The step of obtaining a traffic video training sample set includes: Obtain multiple traffic video clips in various traffic scenarios; Collecting the focus point data of multiple observers in the multiple traffic video clips to obtain the multiple focus point data of each video frame; Generating a saliency heat map of each video frame according to the multiple focus point data of each video frame; A traffic monitoring video sample set is constructed based on the multiple traffic video clips and the saliency heat map of each video frame.

3. The method according to claim 1, characterized in that The method of pre-training the prior knowledge learning subnetwork based on the traffic video training sample set, learning the traffic video prior knowledge, and obtaining a preset video saliency prediction model includes: Inputting the plurality of traffic video clips into the prior knowledge learning subnetwork, learning the traffic video prior knowledge, and obtaining a predicted salient object map of the last video frame of each video clip; Determining a first loss function according to the actual salient object map of each video clip and the predicted salient object map; The weight parameters of the initial video saliency prediction model are adjusted according to the first loss function to obtain a preset video saliency prediction model.

4. The method according to claim 1, characterized in that: Based on the learned prior knowledge of traffic videos, fine-tuning the video saliency prediction subnetwork in the preset video saliency prediction model to obtain a target video saliency prediction model includes: Inputting the plurality of traffic video clips into the preset video saliency prediction model to obtain a predicted saliency heat map of each video frame of each video clip; Determining a second loss function based on the predicted saliency heat map of each video frame and the actual saliency heat map of each video frame; The weight parameters of the preset video saliency prediction model are adjusted according to the second loss function to obtain a target video saliency prediction model.

5. The method according to claim 1, characterized in that After obtaining the traffic video training sample set, the method further includes: For each traffic video clip, target object detection is performed on each video frame based on the target detection model to obtain the target object detection result of each video frame; Based on the target object detection result of each video frame, generating a car distribution map of each video frame; The similarity between the car distribution map of each video frame and the saliency heat map of each video frame is calculated to determine the degree of correlation between video saliency and vehicle distribution.

6. A traffic video saliency prediction method, characterized in that: The method comprises: Acquire a traffic video to be predicted; the traffic video to be predicted includes a plurality of continuous video frames; The traffic video to be predicted is input into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method according to any one of claims 1-5.

7. A traffic video saliency prediction model generation device, characterized in that: The traffic video saliency prediction model generation device comprises: A sample set acquisition module is used to acquire a traffic video training sample set; the traffic video training sample set includes: a plurality of traffic video clips and a saliency heat map of each video frame; wherein each traffic video clip includes a plurality of continuous video frames; A model building module, used to build an initial video saliency prediction model; the initial video saliency prediction model is composed of a priori knowledge learning subnetwork of a shared encoder and a video saliency prediction subnetwork; A pre-training module, used to pre-train the prior knowledge learning sub-network based on the traffic video training sample set, learn the traffic video prior knowledge, and obtain a preset video saliency prediction model; The generation module is used to fine-tune the video saliency prediction subnetwork in the preset video saliency prediction model based on the learned prior knowledge of traffic videos to obtain a target video saliency prediction model.

8. A traffic video saliency prediction device, characterized in that: The traffic video saliency prediction device comprises: A video acquisition module, used to acquire a traffic video to be predicted; the traffic video to be predicted includes a plurality of continuous video frames; A saliency prediction module is used to input the traffic video to be predicted into the target video saliency prediction model to obtain a saliency heat map corresponding to each video frame of the traffic video to be predicted; the target video saliency prediction model is obtained based on the traffic video saliency prediction model generation method described in any one of claims 1-5.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for generating a traffic video significance prediction model according to any one of claims 1 to 5 or the method for predicting traffic video significance according to claim 6 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for generating a traffic video significance prediction model according to any one of claims 1 to 5, or the method for predicting traffic video significance according to claim 6 is implemented.

Images