Traffic scene analysis method and device based on video stream

Abstract

The present application discloses a method and device for analyzing traffic scenes based on video streams. The method includes: inputting multiple key frame images into a backbone network to extract key frame features, and obtaining multiple key frame feature maps; inputting the key frame image with the closest time distance to each non-key frame image and the corresponding key frame feature map and each non-key frame image into a feature mapping network for feature mapping, and obtaining a non-key frame mapping feature map corresponding to each non-key frame image; inputting the key frame feature map and the non-key frame mapping feature map into a semantic segmentation branch network and an instance segmentation branch network for prediction, and obtaining a predicted panoramic segmentation result of the video stream; constructing a spatial feature constraint loss function based on the predicted panoramic segmentation result and the true panoramic segmentation result, and constructing a temporal feature constraint loss function based on each non-key frame feature map and the corresponding non-key frame mapping feature map.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a traffic scene analysis method and device based on video stream. Background Art

[0002] With the development of high-position video technology, data is collected by installing high-position video cameras on the roadside. The collected data can be used for vehicle detection, license plate detection, pedestrian detection, lane line detection, parking line detection, passable area segmentation, roadside green plant segmentation and other tasks using visual algorithms, thereby realizing a full range of traffic scene analysis functions for the entire area monitored by the high-position camera. By analyzing the traffic scene, it is conducive to more refined and accurate management of roadside parking, and can assist in providing more evidence about vehicle violations, irregular driving, etc. Therefore, by analyzing the traffic scene, it is possible to achieve more intelligent application and management of the entire traffic flow, which has a positive role in promoting various aspects of urban traffic management, driving safety, etc.

[0003] Traditional traffic scene analysis methods analyze individual scenes based on single-frame images and fuse the analysis results of multiple single-frame images to achieve traffic scene analysis. However, for video streams, the traditional method of analyzing and fusing single-frame images one by one will result in incoherent traffic scene analysis results, feature information will be discarded, and the traffic scene cannot be accurately analyzed comprehensively.

[0004] Application Contents

[0005] The purpose of this application is to solve the technical problem that the traditional method causes the traffic scene analysis results to be incoherent and cannot accurately and comprehensively analyze the traffic scene. To achieve the above purpose, this application provides a traffic scene analysis method and device based on video stream.

[0006] The present application provides a traffic scene analysis method based on video stream, comprising:

[0007] Acquire a video stream, and perform panoramic segmentation annotation on the video stream to obtain a true panoramic segmentation result of the video stream;

[0008] Extracting key frames from the video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and inputting the plurality of key frame images into the backbone network of the panoramic segmentation model to extract key frame features to obtain a plurality of key frame feature maps;

[0009] Inputting the key frame image closest to each of the non-key frame images in time and the key frame feature map corresponding to the key frame image and each of the non-key frame images into the feature mapping network of the panoramic segmentation model for feature mapping, to obtain a non-key frame mapping feature map corresponding to each of the non-key frame images;

[0010] Inputting the key frame feature map corresponding to the key frame image with the closest temporal distance to each of the non-key frame images and the non-key frame mapping feature map corresponding to each of the non-key frame images into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction, thereby obtaining a predicted panoramic segmentation result of the video stream;

[0011] Constructing a spatial feature constraint loss function according to the predicted panoramic segmentation result and the true panoramic segmentation result, and constructing a temporal feature constraint loss function according to the non-key frame feature map corresponding to each of the non-key frame images and the non-key frame mapping feature map;

[0012] The panoramic segmentation model is trained and optimized according to the spatial feature constraint loss function and the temporal feature constraint loss function to obtain a trained panoramic segmentation model, and traffic scene analysis is performed on the video stream to be tested according to the trained panoramic segmentation model.

[0013] In one embodiment, the step of inputting the key frame image closest to each non-key frame image in time, the key frame feature map corresponding to the key frame image, and each non-key frame image into the feature mapping network of the panoptic segmentation model for feature mapping to obtain the non-key frame mapping feature map corresponding to each non-key frame image includes:

[0014] Inputting the key frame image with the closest time distance to each of the non-key frame images and each of the non-key frame images into an optical flow neural network to obtain a feature optical flow map;

[0015] Feature mapping is performed between each of the feature optical flow maps and the key frame feature map corresponding to the key frame image with the closest time distance, so as to obtain the non-key frame mapping feature map corresponding to each of the non-key frame images.

[0016] In one embodiment, each of the feature optical flow images is feature mapped with the key frame feature image corresponding to the key frame image with the closest time distance to obtain the non-key frame mapping feature image corresponding to each of the non-key frame images, and the non-key frame mapping feature image corresponding to the i-th non-key frame image is represented as f i (p)=G(q,p+δp)f k (q);

[0017] Among them, f i (p) represents the feature at position P in the i-th non-key frame image, f k (q) represents the feature of the kth key frame image at position q that is closest to the i-th non-key frame image in time, G(q,p+δp) represents bilinear interpolation, δp=F i→k (p) represents the position shift when the position q in the k-th key frame image is mapped to the position P in the i-th non-key frame image, and F i→k =f(I k ,I i ) represents the characteristic optical flow map corresponding to the i-th non-key frame image and the k-th key frame image, I k represents the kth key frame image, I i represents the i-th non-key frame image, and f is the optical flow neural network.

[0018] In one embodiment, in constructing a temporal feature constraint loss function according to the non-key frame feature map corresponding to each non-key frame image and the non-key frame mapping feature map, the temporal feature constraint loss function is:

[0019]

[0020] in, represents the non-key frame mapping feature map, y represents the non-key frame feature map, and N represents the number of frames of the image in the video stream.

[0021] In one embodiment, the step of inputting the key frame feature map corresponding to the key frame image with the closest temporal distance to each of the non-key frame images and the non-key frame mapping feature map corresponding to each of the non-key frame image maps into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction to obtain the predicted panoramic segmentation result of the video stream includes:

[0022] Inputting each of the key frame feature maps and each of the non-key frame mapping feature maps into the semantic segmentation branch network of the panoramic segmentation model for semantic prediction, and obtaining a plurality of predicted uncountable target categories of the video stream;

[0023] Inputting each of the key frame feature maps and each of the non-key frame mapping feature maps into the instance segmentation branch network of the panoramic segmentation model for instance prediction, and obtaining a plurality of predicted countable target detection box categories, a plurality of predicted countable target detection box positions, and a plurality of predicted countable target detection box binary masks of the video stream;

[0024] The multiple predicted uncountable target categories are fused with the multiple predicted countable target detection frame categories, the multiple predicted countable target detection frame positions, and the multiple predicted countable target detection frame binary masks to obtain a predicted panoramic segmentation result of the video stream.

[0025] In one embodiment, the video stream is subjected to key frame extraction to obtain a plurality of key frame images and a plurality of non-key frame images, and the plurality of key frame images are input into the backbone network of the panoramic segmentation model to extract key frame features to obtain a plurality of key frame feature maps, including:

[0026] Extracting every K frames in the video stream to obtain the multiple key frame images and the multiple non-key frame images;

[0027] Inputting the multiple key frame images into the backbone network for feature extraction to obtain the multiple key frame feature maps;

[0028] Wherein, K is a positive integer ranging from 3 to 8.

[0029] In one embodiment, the present application provides a traffic scene analysis device based on video stream, comprising:

[0030] A data acquisition module is used to acquire a video stream, and perform panoramic segmentation annotation on the video stream to obtain a true panoramic segmentation result of the video stream;

[0031] A feature extraction module is used to extract key frames from the video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and input the plurality of key frame images into the backbone network of the panoptic segmentation model to extract key frame features to obtain a plurality of key frame feature maps;

[0032] A feature mapping module is used to input the key frame image with the closest time distance to each of the non-key frame images and the key frame feature map corresponding to the key frame image and each of the non-key frame images into the feature mapping network of the panoramic segmentation model for feature mapping, so as to obtain a non-key frame mapping feature map corresponding to each of the non-key frame images;

[0033] A prediction module, configured to input the key frame feature map corresponding to the key frame image with the closest temporal distance to each of the non-key frame images and the non-key frame mapping feature map corresponding to each of the non-key frame images into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction, so as to obtain a predicted panoramic segmentation result of the video stream;

[0034] A loss function construction module, used to construct a spatial feature constraint loss function according to the predicted panoramic segmentation result and the true panoramic segmentation result, and to construct a temporal feature constraint loss function according to the non-key frame feature map corresponding to each of the non-key frame images and the non-key frame mapping feature map;

[0035] The model generation module is used to train and optimize the panoramic segmentation model according to the spatial feature constraint loss function and the temporal feature constraint loss function to obtain a trained panoramic segmentation model, and perform traffic scene analysis on the video stream to be tested according to the trained panoramic segmentation model.

[0036] In one embodiment, the feature mapping module includes:

[0037] A feature optical flow map acquisition module, used for inputting the key frame image with the closest time distance to each of the non-key frame images and each of the non-key frame images into an optical flow neural network to obtain a feature optical flow map;

[0038] The non-key frame mapping feature map acquisition module is used to perform feature mapping between each of the feature optical flow maps and the key frame feature map corresponding to the key frame image with the closest time distance, so as to obtain the non-key frame mapping feature map corresponding to each of the non-key frame images.

[0039] In one embodiment, in the non-key frame mapping feature map acquisition module, the non-key frame mapping feature map corresponding to the i-th non-key frame image is represented by f i (p)=G(q,p+δp)f k (q);

[0040] Among them, f i (p) represents the feature at position P in the i-th non-key frame image, f k (q) represents the feature of the kth key frame image at position q that is closest to the i-th non-key frame image in time, G(q,p+δp) represents bilinear interpolation, δp=F i→k (p) represents the position shift when the position q in the k-th key frame image is mapped to the position P in the i-th non-key frame image, and F i→k =f(I k ,I i ) represents the characteristic optical flow map corresponding to the i-th non-key frame image and the k-th key frame image, I k represents the kth key frame image, I i represents the i-th non-key frame image, and f is the optical flow neural network.

[0041] In one embodiment, in the loss function construction module, the time feature constraint loss function is:

[0042]

[0043] in, represents the non-key frame mapping feature map, y represents the non-key frame feature map, and N represents the number of frames of the image in the video stream.

[0044] In one embodiment, the prediction module includes:

[0045] A semantic segmentation module, used for inputting each of the key frame feature maps and each of the non-key frame mapping feature maps into the semantic segmentation branch network of the panoptic segmentation model for semantic prediction, so as to obtain a plurality of predicted uncountable target categories of the video stream;

[0046] An instance segmentation module is used to input each of the key frame feature maps and each of the non-key frame mapping feature maps into the instance segmentation branch network of the panoramic segmentation model to perform instance prediction, and obtain multiple predicted countable target detection box categories, multiple predicted countable target detection box positions, and multiple predicted countable target detection box binary masks of the video stream;

[0047] A fusion module is used to fuse the multiple predicted uncountable target categories with the multiple predicted countable target detection frame categories, the multiple predicted countable target detection frame positions and the multiple predicted countable target detection frame binary masks to obtain a predicted panoramic segmentation result of the video stream.

[0048] In one embodiment, the feature extraction module includes:

[0049] An image extraction module, used for extracting K frames in the video stream every interval to obtain the multiple key frame images and the multiple non-key frame images;

[0050] A key frame feature map acquisition module, used for inputting the plurality of key frame images into the backbone network for feature extraction to obtain the plurality of key frame feature maps;

[0051] Wherein, K is a positive integer ranging from 3 to 8.

[0052] The above-mentioned traffic scene analysis method and device based on video stream can assign a category to each pixel in the video stream, and can segment the pixel area contained in each instance object, thereby distinguishing all visible content in the view. Through the traffic scene analysis method based on video stream provided by the present application, multiple key frame images and multiple non-key frame images in the video stream can be smoothly connected together, the correlation between each frame image in the video stream is enhanced, and the information between each frame in the video stream is fully integrated, thereby achieving a fine, accurate and smooth and coherent scene analysis of the video stream. Therefore, through the traffic scene analysis method based on video stream provided by the present application, it is possible to realize a full range of traffic scene analysis functions for the entire area monitored by the high-position camera, which is conducive to more precise and accurate management of roadside parking, and improves the management performance in various aspects such as urban traffic management and driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the step flow of the traffic scene analysis method based on video stream provided in this application.

[0054] Figure 2 It is a structural diagram of a traffic scene analysis device based on video stream provided in this application. DETAILED DESCRIPTION

[0055] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments.

[0056] See also Figure 1 The present application provides a traffic scene analysis method based on video stream, comprising:

[0057] S10, acquiring a video stream, and performing panoramic segmentation and annotation on the video stream to obtain a true panoramic segmentation result of the video stream;

[0058] S20, extracting key frames from the video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and inputting the plurality of key frame images into a backbone network of a panoramic segmentation model to extract key frame features to obtain a plurality of key frame feature maps;

[0059] S30, inputting the key frame image closest to each non-key frame image in time, the key frame feature map corresponding to the key frame image, and each non-key frame image into a feature mapping network of a panoramic segmentation model for feature mapping, and obtaining a non-key frame mapping feature map corresponding to each non-key frame image;

[0060] S40, inputting the key frame feature map corresponding to the key frame image with the closest temporal distance to each non-key frame image and the non-key frame mapping feature map corresponding to each non-key frame image into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction, and obtaining a predicted panoramic segmentation result of the video stream;

[0061] S50, constructing a spatial feature constraint loss function according to the predicted panoramic segmentation result and the actual panoramic segmentation result, and constructing a temporal feature constraint loss function according to the non-key frame feature map and the non-key frame mapping feature map corresponding to each non-key frame image;

[0062] S60, training and optimizing the panoramic segmentation model according to the spatial feature constraint loss function and the temporal feature constraint loss function to obtain a trained panoramic segmentation model, and performing traffic scene analysis on the video stream to be tested according to the trained panoramic segmentation model.

[0063] In S10, a traffic scene video stream is obtained according to the high-position video data. The video stream is split into images frame by frame, and the video stream is panoramically segmented and annotated to obtain a true panoramic segmentation result corresponding to the video stream. The high-position video data can be obtained by a high-position video camera. The high-position video camera is a camera installed on the roadside to shoot traffic scene information. The true panoramic segmentation result may include the category of uncountable targets, the category of countable targets, and the instance of each countable target. The category of each uncountable target, the category of each countable target, and the instance of each countable target can be distinguished by assigning different IDs. The category of uncountable targets can be the category corresponding to uncountable targets such as the sky, roads, and green plants. The category of countable targets can be the category corresponding to countable targets such as vehicles, pedestrians, and traffic signs. In one embodiment, when the video stream is panoramically segmented and annotated, the pixel area contained in each uncountable target can be obtained by a polygon annotation tool, and the ID of each uncountable target can be set to 0. When performing panoramic segmentation annotation on a video stream, you can use the polygon annotation tool to obtain the pixel area contained in each countable target, distinguish the pixel areas of different instances of each countable target, and assign a different instance ID to each instance of each countable target. Instances can be understood as countable targets, such as vehicles, pedestrians, lane lines, green plants, non-motor vehicles, traffic signs, etc.

[0064] In S20, feature extraction is performed through a backbone network. The backbone network includes a convolutional neural network structure, such as Visual Geometry Group Network and ResNet (Residual Network, etc. Feature extraction is performed on multiple key frame images in the video stream through the backbone network to obtain multiple key frame feature maps. Key frame extraction is performed on the video stream to obtain multiple key frame images and multiple non-key frame images, and each frame image in the video stream is distinguished and divided.

[0065] In S30, the feature representation of each key frame feature map obtained in S20 is mapped to other non-key frame images. The feature mapping network can be implemented by an optical flow neural network (FlowNet). A non-key frame feature map is feature mapped with a key frame feature map through a feature mapping network. The time distance between a non-key frame image and a key frame image is the shortest, which can be understood as the shortest interval between the two images. Feature mapping is performed through a feature mapping network, so that a non-key frame image with the closest time distance is feature mapped with a key frame image, so as to represent the features of the key frame image in the feature map corresponding to the non-key frame image, and form a corresponding non-key frame mapping feature map. By constructing a corresponding relationship between the key frame image and the non-key frame image with the closest time distance, the correlation between each frame image in the video stream is enhanced, so that the subsequent analysis of the video stream is more coherent, and there will be no problem of feature information discarding.

[0066] In S40, a non-key frame image corresponds to a key frame image with the closest time distance and a non-key frame mapping feature map, and there is a one-to-one correspondence between the three. The non-key frame mapping feature map represents the correlation between the non-key frame image and the key frame image with the closest time distance, and then establishes a time correlation relationship between the frames of the video stream. Therefore, through multiple non-key frame mapping feature maps, the frames in the video stream can be connected, so that the frames can be more coherent. A non-key frame mapping feature map with coherence and a key frame feature map with the closest time distance are input into the semantic segmentation branch network for semantic segmentation, and there will be no problem of losing feature information. A non-key frame mapping feature map with coherence and a key frame feature map with the closest time distance are input into the instance segmentation branch network for instance segmentation, and there will be no problem of losing feature information. By performing semantic segmentation and instance segmentation on the non-key frame mapping feature map and the key frame feature map with the closest time distance, the predicted panoramic segmentation result corresponding to the video stream can be obtained.

[0067] In S50, the spatial feature constraint loss function and the temporal feature constraint loss function form a total loss function, that is, the loss function of the panoramic segmentation model. In S60, the panoramic segmentation model is trained and optimized according to the loss function of the panoramic segmentation model to obtain optimized model parameters, thereby obtaining a stable panoramic segmentation model. According to the trained panoramic segmentation model, the traffic scene of the video stream to be tested can be analyzed to obtain a panoramic segmentation result.

[0068] Through the video stream-based traffic scene analysis method provided by the present application, it is possible to assign a category to each pixel in the video stream, and it is possible to segment the pixel area contained in each instance object, and it is possible to distinguish all visible content in the view. Through the video stream-based traffic scene analysis method provided by the present application, multiple key frame images and multiple non-key frame images in the video stream can be smoothly connected together, the correlation between each frame image in the video stream is enhanced, and the information between each frame in the video stream is fully integrated, thereby achieving a fine, accurate and smooth and coherent scene analysis of the video stream. Therefore, through the video stream-based traffic scene analysis method provided by the present application, it is possible to achieve a full range of traffic scene analysis functions for the entire area monitored by the high-position camera, which is conducive to more precise and accurate management of roadside parking, and improves the management performance in various aspects such as urban traffic management and driving safety.

[0069] In one embodiment, S20, a key frame is extracted from a video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and the plurality of key frame images are input into a backbone network of a panoramic segmentation model to extract key frame features to obtain a plurality of key frame feature maps, including:

[0070] S210, extracting every K frames in the video stream to obtain a plurality of key frame images and a plurality of non-key frame images;

[0071] S220, inputting a plurality of key frame images into a backbone network for feature extraction to obtain a plurality of key frame feature maps;

[0072] Among them, multiple key frame images and multiple non-key frame images form a video stream, and K is a positive integer in the range of 3 to 8.

[0073] In this embodiment, the key frame image is a video frame extracted every K frames in the continuous video frames. A plurality of key frame images are removed from the video stream, and the remaining video frames are all non-key frame images. It can also be understood that a plurality of key frame images and a plurality of non-key frame images together form a video stream. K is a positive integer in the range of 3 to 8, and can be 3, 4, 5, 6, 7, or 8. By setting the value of K, the video stream can be better distinguished between a plurality of key frame images and a plurality of non-key frame images, and the loss of inter-frame correlation due to too many interval frames can be avoided. By performing feature extraction on the video stream every K frames, the traffic scene analysis method of the present application can achieve fine, accurate, smooth and coherent scene analysis of the video stream.

[0074] In one embodiment, S30, the key frame image closest to each non-key frame image in time and the key frame feature map corresponding to the key frame image are input into the feature mapping network of the panoramic segmentation model with each non-key frame image for feature mapping, and a non-key frame mapping feature map corresponding to each non-key frame image is obtained, including:

[0075] S310, inputting the key frame image with the closest time distance to each non-key frame image and each non-key frame image into an optical flow neural network to obtain a feature optical flow map;

[0076] S320, feature mapping is performed between each feature optical flow map and a key frame feature map corresponding to a key frame image with the shortest time distance, to obtain a non-key frame mapping feature map corresponding to each non-key frame image.

[0077] In S310, the input of the optical flow neural network is the current non-key frame image and the key frame image with the closest time distance. The calculation process of the optical flow neural network includes a contraction part composed of a convolution layer, a correlation layer part, and an expansion part composed of a deconvolution layer. The contraction part composed of the convolution layer is used to extract the feature map of the two frames. The correlation layer part is used to calculate the correlation between the two feature maps. The expansion part composed of the deconvolution layer is used to perform optical flow prediction to obtain a feature optical flow map.

[0078] In S320, feature mapping is performed using the feature optical flow map and the key frame feature map with the closest temporal distance to achieve feature alignment of the key frame with the closest temporal distance, and propagate it to the feature map of the corresponding non-key frame image to obtain the output feature map corresponding to the non-key frame image, that is, the non-key frame mapping feature map.

[0079] The non-keyframe mapping feature map reflects the feature fusion relationship between the non-keyframe image and the keyframe image with the closest time distance, so that the time factor is fully considered in the subsequent prediction of the semantic segmentation branch network and the instance segmentation branch network, and the correlation between each frame image in the video stream is enhanced, making the subsequent analysis of the video stream more coherent and there will be no problem of feature information discarding.

[0080] In one embodiment, S320, feature mapping is performed between each feature optical flow map and the key frame feature map corresponding to the key frame image with the closest time distance, and a non-key frame mapping feature map corresponding to each non-key frame image is obtained. The non-key frame mapping feature map corresponding to the i-th non-key frame image is represented as f i (p)=G(q,p+δp)f k (q);

[0081] Among them, f i (p) represents the feature at position P in the i-th non-keyframe image, f k (q) represents the feature of the kth key frame image at position q that is closest to the i-th non-key frame image in time, G(q,p+δp) represents bilinear interpolation, δp = F i→k (p) represents the position shift when the position q in the k-th key frame image is mapped to the position P in the i-th non-key frame image, F i→k =f(I k ,I i ) represents the characteristic optical flow map corresponding to the i-th non-key frame image and the k-th key frame image, I k represents the kth key frame image, I i represents the i-th non-key frame image, and f is the optical flow neural network.

[0082] In this embodiment, feature mapping is performed by bilinear interpolation. In the feature mapping process, the position of the pixel in the non-key frame image in the key frame image with the closest time distance can be obtained based on the key frame feature map corresponding to the known key frame image with the closest time distance and the feature optical flow map corresponding to the non-key frame image. Bilinear interpolation is used to perform feature mapping of the feature optical flow map of each non-key frame image with the key frame feature map of the key frame image with the closest time distance, so that the connection between the non-key frame image and the key frame image with the closest time distance is closer, making the subsequent analysis of the video stream more coherent, and there will be no problem of feature information discarding.

[0083] In one embodiment, S40, the key frame feature map corresponding to the key frame image with the closest time distance to each non-key frame image and the non-key frame mapping feature map corresponding to each non-key frame image are input into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction, and the predicted panoramic segmentation result of the video stream is obtained, including:

[0084] S410, inputting each key frame feature map and each non-key frame mapping feature map into a semantic segmentation branch network of a panoramic segmentation model for semantic prediction, and obtaining multiple predicted uncountable target categories of the video stream;

[0085] S420, inputting each key frame feature map and each non-key frame mapping feature map into the instance segmentation branch network of the panoramic segmentation model for instance prediction, and obtaining multiple predicted countable target detection box categories, multiple predicted countable target detection box positions, and multiple predicted countable target detection box binary masks of the video stream;

[0086] S430, fusing the multiple predicted uncountable target categories with the multiple predicted countable target detection frame categories, the multiple predicted countable target detection frame positions, and the predicted countable target detection frame binary masks to obtain a predicted panoramic segmentation result of the video stream.

[0087] In S410, each key frame feature map and non-key frame mapping feature map are input into the upsampling layer of the semantic segmentation branch network for amplification to obtain multiple amplified feature maps of the video stream. Each amplified feature map is subjected to high- and low-layer feature fusion to obtain multiple fused feature maps of the video stream. According to the normalized exponential function, the multiple fused feature maps of the video stream are classified to obtain multiple predicted uncountable target categories of the video stream. Multiple predicted uncountable target categories can also be understood as multiple predicted pixel categories, which characterize the categories corresponding to uncountable targets such as the sky, roads, and green plants.

[0088] The semantic segmentation branch network mainly includes multiple upsampling operations and feature fusion operations. The features of each keyframe feature map and non-keyframe mapping feature map are upsampled to restore the feature size to the original image size. Finally, the classification probability of each pixel is obtained through the Softmax normalized exponential function, and then multiple predicted uncountable target categories in the video stream are obtained.

[0089] In one embodiment, the semantic segmentation branch network includes but is not limited to the commonly used semantic segmentation networks such as FCN, DeepLab, and U-Net. The upsampling operation includes but is not limited to the use of deconvolution operations, bilinear interpolation operations, etc. to restore the size of the feature map. The high- and low-layer feature fusion interpolates the low-resolution high-level features to be fused to the same resolution as the high-resolution low-level features by bilinear interpolation and other methods, and then performs feature fusion by addition or splicing along the channel dimension. Through the fusion of high- and low-layer features, it is possible to fuse the features between different layers of the network and realize the fusion between high- and low-layer features, thereby improving the segmentation accuracy of semantic segmentation. The fusion of high- and low-layer features includes but is not limited to the use of feature fusion methods such as FPN (Feature Pyramid Network) and ASPP (Atrous Spatial Pooling Pyramid). The Softmax normalized exponential function is a multi-classification calculation function, which converts the prediction result of each point into the probability of classification. The calculation formula is as follows:

[0090]

[0091] Among them, z c Represents the output value, It means that the prediction result is transformed into an exponential function, thus ensuring the non-negativity of the probability. It means that the predicted result after exponential transformation of each pixel is divided by the sum of the results after exponential transformation of all pixels, that is, the percentage of the result after exponential transformation of each pixel in the total number, to obtain the classification probability.

[0092] In S420, multiple feature maps are input into the region extraction network layer to obtain multiple target detection frames. Multiple target detection frames are input into the detection and mask prediction layer to obtain the category, position and binary mask of each target detection frame. By constructing an instance segmentation branch network, instance prediction is achieved. In one embodiment, the instance segmentation branch network can use the Mask R-CNN instance segmentation algorithm as the instance segmentation branch network architecture. Mask R-CNN belongs to a two-stage instance segmentation method. First, each key frame feature map and each non-key frame mapping feature map are input into the region extraction network RPN (Region Proposal Network) to generate multiple candidate region frames ROI (Region Of Interest). Then the obtained multiple candidate region frames are input into the detection and mask prediction layer to generate the category information of each ROI and the position information of its detection frame and the binary mask of each ROI. Through the region extraction network RPN, candidate frames of multiple targets can be generated. The target detection frame can also be understood as a target candidate frame. The candidate frames of the generated multiple targets are adjusted to the same size through the candidate frame alignment layer. The detection and mask prediction layer consists of two branches, where the detection branch predicts the category and position of the candidate region box, and the mask branch performs pixel-by-pixel semantic segmentation of the candidate region box, that is, obtains the binary mask of the ROI. The binary mask means that for the candidate region box, the pixel value of the area belonging to the background is predicted to be 0, and the pixel value of the area belonging to the target instance is predicted to be 1.

[0093] In S430, the output results of the two branches of the semantic segmentation branch network and the instance segmentation branch network are subjected to feature fusion processing to obtain a panoramic segmentation output. When performing feature fusion processing, a heuristic algorithm can be used to resolve pixel points with pixel conflicts and to solve the problem of pixel allocation conflicts.

[0094] In one embodiment, in S50, a total loss function is formed by using a spatial feature constraint loss function and a temporal feature constraint loss function. Model optimization and parameter update are performed by constructing a total loss function for model training.

[0095] In this embodiment, the gap between the predicted panoramic segmentation result and the true panoramic segmentation result is calculated, and the goal is to reduce the gap between the predicted panoramic segmentation result and the true panoramic segmentation result. The temporal feature constraint loss function constrains the features between the key frames and non-key frames in the time series, and the purpose is to calculate the difference between each non-key frame image and the non-key frame mapping feature map corresponding to each predicted non-key frame image, so as to obtain a stable and coherent panoramic video segmentation result as much as possible.

[0096] The total loss function L can be expressed as L = α1Lspatial +α2L temporal .

[0097] Among them, α1 and α2 represent weight coefficients, which can be set to 1, L spatial represents the spatial feature constraint loss function, L temporal represents the time feature constraint loss function.

[0098] In one embodiment, in S50, according to the non-key frame feature map corresponding to each non-key frame image and the non-key frame mapping feature map corresponding to each non-key frame image, a temporal feature constraint loss function is constructed, and the temporal feature constraint loss function is:

[0099]

[0100] in, represents the non-keyframe mapping feature map, y represents the non-keyframe feature map, and N represents the number of frames of the image in the video stream.

[0101] L temporal Represents the temporal feature constraint loss function. The temporal feature constraint loss function adopts the L2 norm loss function. The L2 norm loss function is also called the minimum square error loss function, which means minimizing the sum of the squares of the difference between the target value and the estimated value. By adjusting the temporal feature constraint loss function, the difference between the predicted non-key frame feature map (which can also be understood as the non-key frame mapping feature map corresponding to the non-key frame image) and the real non-key frame feature map can be reduced.

[0102] In one embodiment, the true panoramic segmentation result includes multiple true uncountable target categories, multiple true countable target detection frame categories, multiple true countable target detection frame positions, and multiple true countable target detection frame binary masks.

[0103] In S50, a spatial feature constraint loss function is constructed according to the predicted panoramic segmentation result and the actual panoramic segmentation result, including:

[0104] According to multiple real uncountable target categories and multiple predicted uncountable target categories, a loss function of the semantic segmentation branch network is constructed;

[0105] A countable target detection box classification loss function is constructed based on multiple predicted countable target detection box categories and multiple real countable target detection box categories. A countable target detection box position loss function is constructed based on multiple predicted countable target detection box positions and multiple real countable target detection box positions. A countable mask segmentation loss function is constructed based on multiple predicted countable target detection box binary masks and multiple real countable target detection box binary masks. A loss function of an instance segmentation branch network is constructed based on the countable target detection box classification loss function, the countable target detection box position loss function and the countable mask segmentation loss function.

[0106] According to the loss function of the semantic segmentation branch network and the loss function of the instance segmentation branch network, a spatial feature constraint loss function is constructed.

[0107] In this embodiment, L spatial Represents the spatial feature constraint loss function, including the semantic segmentation branch loss function and the instance segmentation branch loss function, which can be expressed as L spatial =β1L semantic +β2L instance .

[0108] Among them, β1 and β2 represent weight coefficients, which can be set to 1, L semantic represents the semantic segmentation branch loss function, L instance represents the instance segmentation branch loss function.

[0109] L semantic Represents the semantic segmentation branch loss function, which adopts the cross entropy loss function.

[0110] in, C represents the number of categories for semantic segmentation, P c Indicates the probability that the predicted sample belongs to class c, y c Represents the true label category distribution.

[0111] L instance The instance segmentation branch loss function can be expressed as follows:

[0112] L instance =λ1L cls +λ2L box +λ3L mask .

[0113] Among them, λ1, λ2, and λ3 represent weight coefficients and can be set to 1. cls represents the countable target detection box classification loss function, L box represents the countable target detection box position loss function, L mask represents the countable mask segmentation loss function.

[0114] The countable target detection box classification loss function can be expressed as follows:

[0115]

[0116] Among them, N represents the total number of predicted countable target detection boxes, P i Represents the probability of predicting the countable target detection box as the target, Represents the true label corresponding to the predicted countable target detection box.

[0117] In one embodiment, the countable target detection box position loss function adopts an IoU Loss loss function. The IoULoss loss function is based on the IoU intersection-over-union ratio between the predicted countable target detection box position and the actual countable target detection box position.

[0118] In one embodiment, when performing panoptic segmentation annotation on the video stream in S10, a polygon annotation method is used to annotate the specific outline of each instance. An approximate method is used to obtain an annotation box, and the coordinate list (x min ,y min ) are used as the coordinates of the upper left corner and lower right corner of the true countable target detection box, and are recorded as G. The predicted countable target detection box is marked as P, and the corresponding IoU can be expressed as It represents the ratio of the intersection and union of two detection boxes. IoU Loss function L IoU =1-IoU.

[0119] Countable Mask Segmentation Loss Function L mask The binary cross entropy loss function is used for calculation, and the formula is:

[0120]

[0121] Among them, P represents the probability that the predicted countable target detection box binary mask is the true countable target detection box binary mask, and y=0 and y=1 represent the cases where the true countable target detection box binary mask is 0 or 1.

[0122] In one embodiment, S60, the panoramic segmentation model is trained and optimized according to the spatial feature constraint loss function and the temporal feature constraint loss function to obtain a trained panoramic segmentation model, and the traffic scene analysis is performed on the video stream to be tested according to the trained panoramic segmentation model, including:

[0123] The video stream to be tested is obtained and input into the backbone network, feature mapping network, semantic segmentation branch network and instance segmentation branch network of the panoptic segmentation model for prediction, and the panoptic segmentation result corresponding to the video stream to be tested is obtained. The panoptic segmentation result includes the categories of uncountable targets, the categories of countable targets and the instances of each countable target, which can be applied to the roadside parking management system to achieve more refined and accurate management, and improve the management performance in various aspects such as urban traffic management and driving safety.

[0124] See also Figure 2 In one embodiment, the present application provides a traffic scene analysis device 100 based on video stream, including a data acquisition module 10, a feature extraction module 20, a feature mapping module 30, a prediction module 40, a loss function construction module 50 and a model generation module 60.

[0125] The data acquisition module 10 is used to acquire the video stream, and perform panoramic segmentation annotation on the video stream to obtain the real panoramic segmentation result of the video stream. The feature extraction module 20 is used to extract key frames from the video stream, obtain multiple key frame images and multiple non-key frame images, and input the multiple key frame images into the backbone network of the panoramic segmentation model for key frame feature extraction to obtain multiple key frame feature maps. The feature mapping module 30 is used to input the key frame image with the closest time distance to each non-key frame image and the key frame feature map corresponding to the key frame image and each non-key frame image into the feature mapping network of the panoramic segmentation model for feature mapping to obtain the non-key frame mapping feature map corresponding to each non-key frame image. The prediction module 40 is used to input the key frame feature map corresponding to the key frame image with the closest time distance to each non-key frame image and the non-key frame mapping feature map corresponding to each non-key frame image into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction to obtain the predicted panoramic segmentation result of the video stream. The loss function construction module 50 is used to construct a spatial feature constraint loss function according to the predicted panoramic segmentation result and the actual panoramic segmentation result, and to construct a temporal feature constraint loss function according to the non-key frame feature map and the non-key frame mapping feature map corresponding to each non-key frame image. The model generation module 60 is used to train and optimize the panoramic segmentation model according to the spatial feature constraint loss function and the temporal feature constraint loss function, obtain the trained panoramic segmentation model, and perform traffic scene analysis on the video stream to be tested according to the trained panoramic segmentation model.

[0126] In this embodiment, the relevant description of the data acquisition module 10 may refer to the relevant description of S10 in the above embodiment. The relevant description of the feature extraction module 20 may refer to the relevant description of S20 in the above embodiment. The relevant description of the feature mapping module 30 may refer to the relevant description of S30 in the above embodiment. The relevant description of the prediction module 40 may refer to the relevant description of S40 in the above embodiment. The relevant description of the loss function construction module 50 may refer to the relevant description of S50 in the above embodiment. The relevant description of the model generation module 60 may refer to the relevant description of S60 in the above embodiment.

[0127] In one embodiment, the feature mapping module 30 includes a feature optical flow map acquisition module (not labeled in the figure) and a non-keyframe mapping feature map acquisition module (not labeled in the figure).

[0128] The feature optical flow map acquisition module is used to input the key frame image with the closest time distance to each non-key frame image and each non-key frame image into the optical flow neural network to obtain the feature optical flow map. The non-key frame mapping feature map acquisition module is used to perform feature mapping between each feature optical flow map and the key frame feature map corresponding to the key frame image with the closest time distance to obtain the non-key frame mapping feature map corresponding to each non-key frame image.

[0129] In this embodiment, the description of the feature optical flow map acquisition module can refer to the description of S310 in the above embodiment. The description of the non-keyframe mapping feature map acquisition module can refer to the description of S320 in the above embodiment.

[0130] In one embodiment, in the non-key frame mapping feature map acquisition module, the non-key frame mapping feature map corresponding to the i-th non-key frame image is represented as f i (p)=G(q,p+δp)f k (q).

[0131] Among them, f i (p) represents the feature at position P in the i-th non-keyframe image, f k (q) represents the feature of the kth key frame image at position q that is closest to the i-th non-key frame image in time, G(q,p+δp) represents bilinear interpolation, δp = F i→k (p) represents the position shift when the position q in the k-th key frame image is mapped to the position P in the i-th non-key frame image, F i→k =f(I k ,I i ) represents the characteristic optical flow map corresponding to the i-th non-key frame image and the k-th key frame image, I k represents the kth key frame image, I i represents the i-th non-key frame image, and f is the optical flow neural network.

[0132] For the relevant description in this embodiment, reference may be made to the relevant description about S320 in the above embodiment.

[0133] In one embodiment, in the loss function construction module 50, the time feature constraint loss function is:

[0134]

[0135] in, represents the non-keyframe mapping feature map, y represents the non-keyframe feature map, and N represents the number of frames of the image in the video stream.

[0136] In this embodiment, the relevant description about the time feature constraint loss function can refer to the relevant description in the above embodiment.

[0137] In one embodiment, the prediction module 40 includes a semantic segmentation module (not labeled in the figure), an instance segmentation module (not labeled in the figure), and a fusion module (not labeled in the figure).

[0138] The semantic segmentation module is used to input each key frame feature map and each non-key frame mapping feature map into the semantic segmentation branch network of the panoramic segmentation model for semantic prediction, and obtain multiple predicted uncountable target categories of the video stream. The instance segmentation module is used to input each key frame feature map and each non-key frame mapping feature map into the instance segmentation branch network of the panoramic segmentation model for instance prediction, and obtain multiple predicted countable target detection box categories, multiple predicted countable target detection box positions, and multiple predicted countable target detection box binary masks of the video stream.

[0139] The fusion module is used to fuse multiple predicted uncountable target categories with multiple predicted countable target detection frame categories, multiple predicted countable target detection frame positions and multiple predicted countable target detection frame binary masks to obtain the predicted panoramic segmentation result of the video stream.

[0140] In this embodiment, the description of the semantic segmentation module can refer to the description of S410 in the above embodiment. The description of the instance segmentation module can refer to the description of S420 in the above embodiment. The description of the fusion module can refer to the description of S430 in the above embodiment.

[0141] In one embodiment, the feature extraction module 20 includes an image extraction module (not labeled in the figure) and a key frame feature map acquisition module (not labeled in the figure).

[0142] The image extraction module is used to extract every K frames in the video stream to obtain multiple key frame images and multiple non-key frame images. The key frame feature map acquisition module is used to input multiple key frame images into the backbone network for feature extraction to obtain multiple key frame feature maps.

[0143] Wherein, K is a positive integer ranging from 3 to 8.

[0144] In this embodiment, the relevant description of the key frame feature map acquisition module can refer to the relevant description of S210 in the above embodiment.

[0145] In the above-mentioned various embodiments, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims provide the elements of various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0146] Those skilled in the art may also understand that the various illustrative logical blocks, modules and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, modules and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0147] The various illustrative logic blocks described in the embodiments of the present application can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or the design of any combination of the above to implement or operate the described functions. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration to implement.

[0148] The steps of the method or algorithm described in the embodiments of the present application can be directly embedded in the software module executed by the hardware, the processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0149] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A traffic scene analysis method based on video stream, characterized in that: include: Acquire a video stream, and perform panoramic segmentation annotation on the video stream to obtain a true panoramic segmentation result of the video stream; Extracting key frames from the video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and inputting the plurality of key frame images into the backbone network of the panoramic segmentation model to extract key frame features to obtain a plurality of key frame feature maps; Inputting the key frame image closest to each of the non-key frame images in time and the key frame feature map corresponding to the key frame image and each of the non-key frame images into the feature mapping network of the panoramic segmentation model for feature mapping, to obtain a non-key frame mapping feature map corresponding to each of the non-key frame images; Inputting the key frame feature map corresponding to the key frame image with the closest temporal distance to each of the non-key frame images and the non-key frame mapping feature map corresponding to each of the non-key frame images into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction, thereby obtaining a predicted panoramic segmentation result of the video stream; Constructing a spatial feature constraint loss function according to the predicted panoramic segmentation result and the true panoramic segmentation result, and constructing a temporal feature constraint loss function according to the non-key frame feature map corresponding to each of the non-key frame images and the non-key frame mapping feature map; The panoramic segmentation model is trained and optimized according to the spatial feature constraint loss function and the temporal feature constraint loss function to obtain a trained panoramic segmentation model, and traffic scene analysis is performed on the video stream to be tested according to the trained panoramic segmentation model.

2. The method for analyzing traffic scenes based on video stream according to claim 1, characterized in that: The step of inputting the key frame image closest to each non-key frame image in time, the key frame feature map corresponding to the key frame image, and each non-key frame image into the feature mapping network of the panoptic segmentation model for feature mapping to obtain the non-key frame mapping feature map corresponding to each non-key frame image comprises: Inputting the key frame image with the closest time distance to each of the non-key frame images and each of the non-key frame images into an optical flow neural network to obtain a feature optical flow map; Feature mapping is performed between each of the feature optical flow maps and the key frame feature map corresponding to the key frame image with the closest time distance, so as to obtain the non-key frame mapping feature map corresponding to each of the non-key frame images.

3. The method for analyzing traffic scenes based on video stream according to claim 2, characterized in that: The feature mapping is performed on each of the feature optical flow images and the key frame feature map corresponding to the key frame image with the closest time distance, to obtain the non-key frame mapping feature map corresponding to each of the non-key frame images, wherein the non-key frame mapping feature map corresponding to the i-th non-key frame image is represented as f i (p)=G(q,p+δp)f k (q); Among them, f i (p) represents the feature at position P in the i-th non-key frame image, f k (q) represents the feature of the kth key frame image at position q that is closest to the i-th non-key frame image in time, G(q,p+δp) represents bilinear interpolation, δp=F i→k (p) represents the position shift when the position q in the k-th key frame image is mapped to the position P in the i-th non-key frame image, and F i→k =f(I k ,I i ) represents the characteristic optical flow map corresponding to the i-th non-key frame image and the k-th key frame image, I k represents the kth key frame image, I i represents the i-th non-key frame image, and f is the optical flow neural network.

4. The method for analyzing traffic scenes based on video stream according to claim 3 is characterized in that: In constructing a time feature constraint loss function according to the non-key frame feature map corresponding to each non-key frame image and the non-key frame mapping feature map, the time feature constraint loss function is: in, represents the non-key frame mapping feature map, y represents the non-key frame feature map, and N represents the number of frames of the image in the video stream.

5. The method for analyzing traffic scenes based on video stream according to claim 1, characterized in that: The step of inputting the key frame feature map corresponding to the key frame image with the closest time distance to each of the non-key frame images and the non-key frame mapping feature map corresponding to each of the non-key frame images into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction to obtain the predicted panoramic segmentation result of the video stream, comprises: Inputting each of the key frame feature maps and each of the non-key frame mapping feature maps into the semantic segmentation branch network of the panoramic segmentation model for semantic prediction, and obtaining a plurality of predicted uncountable target categories of the video stream; Inputting each of the key frame feature maps and each of the non-key frame mapping feature maps into the instance segmentation branch network of the panoramic segmentation model for instance prediction, and obtaining a plurality of predicted countable target detection box categories, a plurality of predicted countable target detection box positions, and a plurality of predicted countable target detection box binary masks of the video stream; The multiple predicted uncountable target categories are fused with the multiple predicted countable target detection frame categories, the multiple predicted countable target detection frame positions, and the multiple predicted countable target detection frame binary masks to obtain a predicted panoramic segmentation result of the video stream.

6. The method for analyzing traffic scenes based on video stream according to claim 1, characterized in that: The step of extracting key frames from the video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and inputting the plurality of key frame images into the backbone network of the panoramic segmentation model to extract key frame features to obtain a plurality of key frame feature maps includes: Extracting every K frames in the video stream to obtain the multiple key frame images and the multiple non-key frame images; Inputting the multiple key frame images into the backbone network for feature extraction to obtain the multiple key frame feature maps; Wherein, K is a positive integer ranging from 3 to 8.

7. A traffic scene analysis device based on video stream, characterized in that: include: A data acquisition module is used to acquire a video stream, and perform panoramic segmentation annotation on the video stream to obtain a true panoramic segmentation result of the video stream; A feature extraction module is used to extract key frames from the video stream to obtain a plurality of key frame images and a plurality of non-key frame images, and input the plurality of key frame images into the backbone network of the panoptic segmentation model to extract key frame features to obtain a plurality of key frame feature maps; A feature mapping module is used to input the key frame image with the closest time distance to each of the non-key frame images and the key frame feature map corresponding to the key frame image and each of the non-key frame images into the feature mapping network of the panoramic segmentation model for feature mapping, so as to obtain a non-key frame mapping feature map corresponding to each of the non-key frame images; A prediction module, configured to input the key frame feature map corresponding to the key frame image with the closest temporal distance to each of the non-key frame images and the non-key frame mapping feature map corresponding to each of the non-key frame images into the semantic segmentation branch network and the instance segmentation branch network of the panoramic segmentation model for prediction, so as to obtain a predicted panoramic segmentation result of the video stream; A loss function construction module, used to construct a spatial feature constraint loss function according to the predicted panoramic segmentation result and the true panoramic segmentation result, and to construct a temporal feature constraint loss function according to the non-key frame feature map corresponding to each of the non-key frame images and the non-key frame mapping feature map; The model generation module is used to train and optimize the panoramic segmentation model according to the spatial feature constraint loss function and the temporal feature constraint loss function to obtain a trained panoramic segmentation model, and perform traffic scene analysis on the video stream to be tested according to the trained panoramic segmentation model.

8. The traffic scene analysis device based on video stream according to claim 7, characterized in that: The feature mapping module comprises: A feature optical flow map acquisition module, used for inputting the key frame image with the closest time distance to each of the non-key frame images and each of the non-key frame images into an optical flow neural network to obtain a feature optical flow map; The non-key frame mapping feature map acquisition module is used to perform feature mapping between each of the feature optical flow maps and the key frame feature map corresponding to the key frame image with the closest time distance, so as to obtain the non-key frame mapping feature map corresponding to each of the non-key frame images.

9. The method for analyzing traffic scenes based on video stream according to claim 8, characterized in that: In the non-key frame mapping feature map acquisition module, the non-key frame mapping feature map corresponding to the i-th non-key frame image is represented by f i (p)=G(q,p+δp)f k (q); Among them, f i (p) represents the feature at position P in the i-th non-key frame image, f k (q) represents the feature of the kth key frame image at position q that is closest to the i-th non-key frame image in time, G(q,p+δp) represents bilinear interpolation, δp=F i→k (p) represents the position shift when the position q in the k-th key frame image is mapped to the position P in the i-th non-key frame image, and F i→k =f(I k ,I i ) represents the characteristic optical flow map corresponding to the i-th non-key frame image and the k-th key frame image, I k represents the kth key frame image, I i represents the i-th non-key frame image, and f is the optical flow neural network.

10. The method for analyzing traffic scenes based on video stream according to claim 9, characterized in that: In the loss function construction module, the time feature constraint loss function is: in, represents the non-key frame mapping feature map, y represents the non-key frame feature map, and N represents the number of frames of the image in the video stream.

11. The method for analyzing traffic scenes based on video stream according to claim 7, characterized in that: The prediction module comprises: A semantic segmentation module, used for inputting each of the key frame feature maps and each of the non-key frame mapping feature maps into the semantic segmentation branch network of the panoptic segmentation model for semantic prediction, so as to obtain a plurality of predicted uncountable target categories of the video stream; An instance segmentation module is used to input each of the key frame feature maps and each of the non-key frame mapping feature maps into the instance segmentation branch network of the panoramic segmentation model to perform instance prediction, and obtain multiple predicted countable target detection box categories, multiple predicted countable target detection box positions, and multiple predicted countable target detection box binary masks of the video stream; A fusion module is used to fuse the multiple predicted uncountable target categories with the multiple predicted countable target detection frame categories, the multiple predicted countable target detection frame positions and the multiple predicted countable target detection frame binary masks to obtain a predicted panoramic segmentation result of the video stream.

12. The method for analyzing traffic scenes based on video stream according to claim 7, characterized in that: The feature extraction module comprises: An image extraction module, used for extracting K frames in the video stream every interval to obtain the multiple key frame images and the multiple non-key frame images; A key frame feature map acquisition module, used for inputting the plurality of key frame images into the backbone network for feature extraction to obtain the plurality of key frame feature maps; Wherein, K is a positive integer ranging from 3 to 8.

Images