Video semantic segmentation method and system based on monitoring scene

Abstract

The application discloses a video semantic segmentation method and system based on a monitoring scene, and relates to the technical field of image processing. The method comprises the following steps: performing feature extraction on each key frame image to obtain a key frame feature map; performing resolution compression on each non-key frame image to obtain a compressed resolution image, and performing feature extraction on the compressed resolution image to obtain a compressed non-key frame feature map; performing up-sampling processing on the compressed non-key frame feature map to obtain a first non-key frame feature map; performing optical flow mapping transformation on each key frame feature map to obtain a second non-key frame feature map; performing pixel-by-pixel addition on the first non-key frame feature map and the second non-key frame feature map to obtain a fused non-key frame feature map; and performing frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map to obtain a semantic segmentation result. Through the method, fast video semantic segmentation is realized, the calculation of computing power is reduced, and the efficiency of video semantic segmentation is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video semantic segmentation method and system based on surveillance scenarios. Background Technology

[0002] In recent years, high-position video technology has developed rapidly. By installing video surveillance cameras on the roadside to collect data, visual algorithms can be used to perform various tasks such as vehicle detection, license plate recognition, vehicle tracking, pedestrian detection, and parking line detection. This is conducive to more refined and accurate management of roadside parking. In addition, it can provide data for the deployment and development of vehicle-road cooperative technology, thus playing a positive role in promoting various aspects of urban traffic management and driving safety.

[0003] Semantic segmentation, in particular, involves classifying each pixel in an image. The image resolution is crucial to the segmentation accuracy and efficiency of this task. However, traditional methods are limited by computational power, resulting in low segmentation efficiency and an inability to achieve real-time performance. This can lead to the loss of keyframe information during high-resolution video scene analysis, resulting in false alarms in tasks such as roadside parking space management. Summary of the Invention

[0004] The purpose of this invention is to address the technical problem of low segmentation efficiency caused by the limitation of computing power in traditional methods. To achieve the above objective, this invention provides a video semantic segmentation method and system based on surveillance scenarios.

[0005] This invention provides a video semantic segmentation method based on surveillance scenarios, comprising:

[0006] Obtain the keyframe image set and the non-keyframe image set;

[0007] Feature extraction is performed on each of the keyframe images to obtain a keyframe feature map;

[0008] Each of the non-keyframe images is compressed to obtain a compressed resolution image, and features are extracted from the compressed resolution image to obtain a compressed non-keyframe feature map.

[0009] The compressed non-key frame feature map is upsampled to obtain the first non-key frame feature map;

[0010] Perform optical flow mapping transformation on each of the key frame feature maps to obtain a second non-key frame feature map;

[0011] The first non-key frame feature map and the second non-key frame feature map are added pixel by pixel to obtain a fused non-key frame feature map;

[0012] The fused non-keyframe feature map and the keyframe feature map are subjected to frame-by-frame semantic segmentation to obtain semantic segmentation results.

[0013] In one embodiment, obtaining the keyframe image set and the non-keyframe image set includes:

[0014] Multiple video clips are acquired, and each of the monitored video clips is semantically segmented, labeled, and divided into frames to form the key frame image set and the non-key frame image set.

[0015] The number of non-keyframe image sets is N times the number of keyframe image sets, where N is a positive integer greater than 10.

[0016] In one embodiment, the step of extracting features from each of the keyframe images to obtain a keyframe feature map includes:

[0017] Feature extraction is performed on each keyframe image using a first deep residual network to obtain the keyframe feature map;

[0018] The step of extracting features from the compressed resolution image to obtain a compressed non-keyframe feature map includes:

[0019] The compressed resolution image is feature extracted using a second deep residual network to obtain the compressed non-keyframe feature map.

[0020] The backbone network structure of the first deep residual network is larger than that of the backbone network of the second deep residual network.

[0021] In one embodiment, the upsampling process performed on the compressed non-key frame feature map to obtain a first non-key frame feature map includes:

[0022] The first non-key frame feature map is obtained by upsampling the compressed non-key frame feature map using the deconvolution module.

[0023] The step of performing frame-by-frame semantic segmentation on the fused non-keyframe feature map and the keyframe feature map to obtain semantic segmentation results includes:

[0024] The semantic segmentation result is obtained by performing frame-by-frame semantic segmentation on the fused non-keyframe feature map and the keyframe feature map using a fully convolutional multilayer neural network.

[0025] In one embodiment, after performing frame-by-frame semantic segmentation on the fused non-keyframe feature map and the keyframe feature map using a fully convolutional multilayer neural network to obtain the semantic segmentation result, the method further includes:

[0026] Construct a target loss function based on the semantic segmentation annotations and the semantic segmentation results;

[0027] The fully convolutional multilayer neural network is trained and optimized based on the target loss function.

[0028] This invention provides a video semantic segmentation system based on surveillance scenarios, characterized in that it includes:

[0029] The data acquisition module is used to acquire keyframe image sets and non-keyframe image sets;

[0030] The keyframe feature extraction module is used to extract features from each of the keyframe images to obtain a keyframe feature map.

[0031] The compression feature extraction module is used to compress the resolution of each of the non-keyframe images to obtain a compressed resolution image, and to extract features from the compressed resolution image to obtain a compressed non-keyframe feature map.

[0032] The recovery module is used to upsample the compressed non-key frame feature map to obtain the first non-key frame feature map.

[0033] The mapping module is used to perform optical flow mapping transformation on each of the key frame feature maps to obtain a second non-key frame feature map.

[0034] The fusion module is used to add the first non-key frame feature map and the second non-key frame feature map pixel by pixel to obtain a fused non-key frame feature map.

[0035] The semantic segmentation module is used to perform frame-by-frame semantic segmentation on the fused non-keyframe feature map and the keyframe feature map to obtain semantic segmentation results.

[0036] In one embodiment, the data acquisition module is used to acquire multiple video segments, perform semantic segmentation and frame division on each of the monitored video segments, and form the key frame image set and the non-key frame image set.

[0037] The number of non-keyframe image sets is N times the number of keyframe image sets, where N is a positive integer greater than 10.

[0038] In one embodiment, the keyframe feature extraction module is further configured to perform feature extraction on each keyframe image according to the first deep residual network to obtain the keyframe feature map;

[0039] The compression feature extraction module is further used to extract features from the compressed resolution image based on the second deep residual network to obtain the compressed non-keyframe feature map;

[0040] The backbone network structure of the first deep residual network is larger than that of the backbone network of the second deep residual network.

[0041] In one embodiment, the recovery module is further configured to perform upsampling processing on the compressed non-key frame feature map according to the deconvolution module to obtain the first non-key frame feature map;

[0042] The semantic segmentation module is further configured to perform frame-by-frame semantic segmentation on the fused non-keyframe feature map and the keyframe feature map using a fully convolutional multilayer neural network to obtain the semantic segmentation result.

[0043] In one embodiment, the system further includes:

[0044] The training objective construction module is used to construct an objective loss function based on the semantic segmentation annotations and the semantic segmentation results;

[0045] The training module is used to train and optimize the fully convolutional multilayer neural network according to the target loss function.

[0046] The aforementioned video semantic segmentation method and system based on surveillance scenarios divides keyframe images into keyframe images and non-keyframe images. The non-keyframe images are then subjected to resolution compression, feature extraction, and upsampling processing sequentially. This allows semantic segmentation to be performed on non-keyframe images using compressed resolution images, while semantic segmentation is only performed on keyframe images using the original resolution images. This reduces the computational cost of segmentation using only the original resolution, improving computational efficiency and enabling real-time segmentation. The first non-keyframe feature map formed after resolution compression is added pixel-by-pixel to the second non-keyframe feature map formed after optical flow mapping transformation. This increases feature fusion between images of different resolutions, enabling more accurate semantic parsing of surveillance scene videos. Therefore, the video semantic segmentation method based on surveillance scenarios provided by this invention achieves fast video semantic segmentation, reduces computational cost, improves efficiency, and ensures semantic segmentation accuracy through feature fusion, thus improving the consistency and accuracy of the video semantic segmentation task. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the steps of the video semantic segmentation method based on a surveillance scenario provided by the present invention.

[0048] Figure 2 This is a schematic diagram of the video semantic segmentation system based on a surveillance scenario provided by the present invention. Detailed Implementation

[0049] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0050] Please see Figure 1 This invention provides a video semantic segmentation method based on a surveillance scenario, comprising:

[0051] S10, acquire the keyframe image set and the non-keyframe image set;

[0052] S20, extract features from each keyframe image to obtain keyframe feature maps;

[0053] S30, each non-keyframe image is compressed to obtain a compressed resolution image, and features are extracted from the compressed resolution image to obtain a compressed non-keyframe feature map.

[0054] S40, Upsample the compressed non-key frame feature map to obtain the first non-key frame feature map;

[0055] S50, perform optical flow mapping transformation on each keyframe feature map to obtain the second non-keyframe feature map;

[0056] S60, the first non-key frame feature map and the second non-key frame feature map are added pixel by pixel to obtain the fused non-key frame feature map.

[0057] S70, perform frame-by-frame semantic segmentation on the fused non-keyframe feature map and keyframe feature map to obtain semantic segmentation results.

[0058] In this embodiment, the image frames in a video segment are divided into keyframes and non-keyframes, forming a keyframe image set and a non-keyframe image set. Both sets include semantic segmentation annotations, using targets such as vehicles, pedestrians, parking lines, lane lines, and vegetation as the objects of segmentation annotation. For feature extraction of each keyframe image, a deep residual network can be used to obtain a keyframe feature map. The non-keyframe images have the same size as the keyframe images. Feature extraction is performed on the keyframe images at their original size. The non-keyframe images are then compressed to reduce their size, and feature extraction is performed on the compressed resolution images to obtain compressed non-keyframe feature maps. Finally, the compressed non-keyframe feature maps are upsampled to restore their original size, resulting in the first non-keyframe feature map.

[0059] Optical flow mapping transformation can employ an optical flow estimation network. Based on the optical flow between keyframes and non-keyframes, the features extracted from keyframes undergo an optical flow-based mapping transformation to obtain feature maps for non-keyframes. Optical flow can be understood as the displacement of each pixel from a keyframe to a non-keyframe. Based on the pixel offset from keyframes to non-keyframes in the flow field, the keyframe feature maps are subjected to the same pixel offset to obtain feature maps for non-keyframes, forming a second non-keyframe feature map. In one embodiment, the input to the optical flow estimation network is two adjacent image frames, which are stitched together along the channel dimension. The optical flow estimation network includes an encoder and a decoder. Feature propagation is performed between the optical flow estimation network and the keyframe feature maps to obtain feature maps for non-keyframes with the same resolution as the keyframes, i.e., the second non-keyframe feature map.

[0060] The second non-keyframe feature map is the same size as the first non-keyframe feature map. The upsampled first non-keyframe feature map and the second non-keyframe feature map obtained after optical flow mapping are added pixel-by-pixel to obtain a fused non-keyframe feature map. The fused non-keyframe feature map and the keyframe feature map are then input into the semantic segmentation module for frame-by-frame semantic segmentation to obtain the corresponding semantic segmentation results.

[0061] The video semantic segmentation method based on surveillance scenes provided by this invention divides keyframe images into keyframe images and non-keyframe images. The non-keyframe images are then subjected to resolution compression, feature extraction, and upsampling processing sequentially. This allows semantic segmentation to be performed on non-keyframe images using compressed resolution images, while semantic segmentation is performed only on keyframes using original resolution images. This reduces the computational cost of segmentation using only original resolution images, improving computational efficiency and enabling real-time segmentation. The first non-keyframe feature map formed after resolution compression is added pixel-by-pixel to the second non-keyframe feature map formed after optical flow mapping transformation. This enhances feature fusion between images of different resolutions, enabling more accurate semantic analysis of surveillance scene videos.

[0062] Therefore, the video semantic segmentation method based on monitoring scenarios provided by this invention achieves fast video semantic segmentation, reduces computational power, improves the efficiency of video semantic segmentation, and ensures the accuracy of semantic segmentation through feature fusion, thereby improving the coherence and accuracy of video semantic segmentation tasks.

[0063] In one embodiment, S10, obtaining the keyframe image set and the non-keyframe image set includes:

[0064] S110: Acquire multiple video clips, perform semantic segmentation and frame division on each monitoring video clip, and form a key frame image set and a non-key frame image set;

[0065] The number of non-keyframe image sets is N times the number of keyframe image sets, where N is a positive integer greater than 10.

[0066] In this embodiment, image frames within a continuous time period t are considered as a single video segment. In one embodiment, the continuous time period t can be set to 5 to 8 minutes. Video data of the monitored scene is captured using roadside video surveillance cameras, and divided into multiple video segments using the continuous time period t as the dividing point. The monitored scene encompasses data from different camera installation locations, different monitoring angles, and different weather conditions. By utilizing information between consecutive video frames, more coherent and accurate segmentation results can be achieved. Furthermore, by using segmentation to analyze targets such as vehicles and pedestrians, compared to the detection boxes obtained from two-dimensional target detection, the segmentation provides a more detailed and comprehensive view of the target's shape, offering a more accurate basis for judgment.

[0067] Semantic segmentation and annotation are performed on each surveillance video segment. Vehicles, pedestrians, parking lines, lane lines, and vegetation are used as the objects for segmentation and annotation. A polygon annotation tool is used to semantically segment and annotate the target objects in each image frame of the video segment, obtaining a semantic segmentation label for each image frame in each video segment. The image frames in each surveillance video segment are divided into keyframes and non-keyframes. Every N frames are defined as keyframes, and the rest are non-keyframes. The number of non-keyframe images is N times the number of keyframe images, where N is greater than 10. Different multiples are used for keyframes and non-keyframes. Semantic segmentation is performed only on keyframes using the original resolution image, while on non-keyframes using compressed resolution images. This reduces computational complexity and improves the efficiency of video semantic segmentation.

[0068] In one embodiment, S20, feature extraction is performed on each keyframe image to obtain a keyframe feature map, including:

[0069] S210, Feature extraction is performed on each keyframe image based on the first deep residual network to obtain the keyframe feature map.

[0070] In this embodiment, a first deep residual network is used as the backbone network for keyframes. The input image size of the first deep residual network is the resolution size of the original video, defined as H*W*C, representing the image height, width, and number of channels, respectively. The number of keyframes is 1 / N of the number of non-keyframes, which is relatively small. The network structure of the first deep residual network is larger than that of the second deep residual network. The first deep residual network can effectively extract features from keyframes and plays an important role in the subsequent feature fusion of keyframes and non-keyframes.

[0071] The first deep residual network can be ResNet-50, with added feature aggregation to aggregate high-level and low-level features extracted by the network. This fusion of features from different layers allows for more thorough feature extraction and improves the accuracy of subsequent semantic segmentation. Feature fusion operations include, but are not limited to, using feature fusion methods such as FPN (Feature Pyramid Network) and ASPP (Atrous Spatial Pooling Pyramid).

[0072] In one embodiment, S30, feature extraction is performed on the compressed resolution image to obtain a compressed non-keyframe feature map, including:

[0073] S310, Feature extraction is performed on the compressed resolution image based on the second deep residual network to obtain a compressed non-keyframe feature map;

[0074] Among them, the backbone network structure of the first deep residual network is larger than that of the backbone network of the second deep residual network.

[0075] In this embodiment, a second deep residual network is used for the backbone network of non-keyframes. The input image size is the resolution size of the original video after compression, defined as (H / 2)*(W / 2)*C. The number of non-keyframes is N times that of keyframes, which is relatively large. The network structure of the second deep residual network is smaller than that of the backbone network of the first deep residual network, thereby reducing the algorithm's overhead and improving the efficiency of video semantic segmentation. The second deep residual network uses ResNet-18.

[0076] In one embodiment, S40, upsampling is performed on the compressed non-key frame feature map to obtain a first non-key frame feature map, including:

[0077] S410: The compressed non-key frame feature map is upsampled by the deconvolution module to obtain the first non-key frame feature map.

[0078] In this embodiment, when upsampling the compressed non-key frame feature map, a deconvolution module is used to implement the upsampling operation, and the stride in the deconvolution is set to 2. By upsampling the compressed non-key frame feature map through the deconvolution module, the compressed non-key frame feature map can be restored to the same size as the key frame feature map, forming the first non-key frame feature map.

[0079] In one embodiment, S70, semantic segmentation is performed frame-by-frame on the fused non-keyframe feature map and keyframe feature map to obtain semantic segmentation results, including:

[0080] S710 performs frame-by-frame semantic segmentation on the fused non-keyframe feature maps and keyframe feature maps using a fully convolutional multilayer neural network to obtain semantic segmentation results.

[0081] In this embodiment, the fully convolutional multilayer neural network includes, but is not limited to, commonly used segmentation networks such as UNet and PSPNet. The first non-keyframe feature map, formed after resolution compression, and the second non-keyframe feature map, formed after optical flow mapping transformation, are added pixel-by-pixel to obtain a fused non-keyframe feature map, enhancing feature fusion between images of different resolutions. The fused non-keyframe feature map and the keyframe feature map are then input into the fully convolutional multilayer neural network for frame-by-frame semantic segmentation, enabling more accurate semantic analysis of surveillance scene videos. The method provided in this embodiment ensures the accuracy of semantic segmentation through feature fusion, improves the coherence and accuracy of video semantic segmentation tasks, and yields more accurate semantic segmentation results.

[0082] In one embodiment, S70, after performing frame-by-frame semantic segmentation on the fused non-keyframe feature map and keyframe feature map using a fully convolutional multilayer neural network to obtain the semantic segmentation result, the method further includes:

[0083] S80, construct the target loss function based on semantic segmentation annotations and semantic segmentation results;

[0084] S90 trains and optimizes the fully convolutional multilayer neural network based on the target loss function.

[0085] In this embodiment, the objective loss function is to reduce the gap between the segmented labels of the input video image and the network output predicted by the model, that is, to reduce the gap between the semantic segmentation labels and the semantic segmentation results. The objective loss function can be the cross-entropy loss function. By setting the objective loss function, the fully convolutional multilayer neural network is trained and optimized to obtain a fully trained fully convolutional multilayer neural network.

[0086] Please see Figure 2This invention provides a video semantic segmentation system 100 based on a surveillance scene. The video semantic segmentation system 100 based on a surveillance scene includes a data acquisition module 10, a keyframe feature extraction module 20, a compressed feature extraction module 30, a restoration module 40, a mapping module 50, a fusion module 60, and a semantic segmentation module 70. The data acquisition module 10 is used to acquire a set of keyframe images and a set of non-keyframe images. The keyframe feature extraction module 20 is used to extract features from each keyframe image to obtain a keyframe feature map. The compressed feature extraction module 30 is used to compress the resolution of each non-keyframe image to obtain a compressed resolution image, and then extract features from the compressed resolution image to obtain a compressed non-keyframe feature map. The restoration module 40 is used to upsample the compressed non-keyframe feature map to obtain a first non-keyframe feature map. The mapping module 50 is used to perform optical flow mapping transformation on each keyframe feature map to obtain a second non-keyframe feature map. The fusion module 60 is used to add the first and second non-keyframe feature maps pixel-by-pixel to obtain a fused non-keyframe feature map. The semantic segmentation module 70 is used to perform frame-by-frame semantic segmentation on the fused non-keyframe feature map and keyframe feature map to obtain semantic segmentation results.

[0087] In this embodiment, the description of the data acquisition module 10 can be referred to the description of S10 in the above embodiment. The description of the keyframe feature extraction module 20 can be referred to the description of S20 in the above embodiment. The description of the compressed feature extraction module 30 can be referred to the description of S30 in the above embodiment. The description of the recovery module 40 can be referred to the description of S40 in the above embodiment. The description of the mapping module 50 can be referred to the description of S50 in the above embodiment. The description of the fusion module 60 can be referred to the description of S60 in the above embodiment. The description of the semantic segmentation module 70 can be referred to the description of S70 in the above embodiment.

[0088] In one embodiment, the data acquisition module is used to acquire multiple video segments, perform semantic segmentation and annotation and frame division on each monitoring video segment, and form a key frame image set and a non-key frame image set.

[0089] The number of non-keyframe image sets is N times the number of keyframe image sets, where N is a positive integer greater than 10.

[0090] In this embodiment, the relevant description of the data acquisition module can be referred to the description of S110 in the above embodiment.

[0091] In one embodiment, the keyframe feature extraction module is further configured to extract features from each keyframe image based on the first deep residual network to obtain a keyframe feature map.

[0092] The compressed feature extraction module is also used to extract features from the compressed resolution image based on the second deep residual network to obtain compressed non-keyframe feature maps;

[0093] Among them, the backbone network structure of the first deep residual network is larger than that of the backbone network of the second deep residual network.

[0094] In this embodiment, the description of the keyframe feature extraction module can be found in the description of S210 in the above embodiment. The description of the compressed feature extraction module can be found in the description of S310 in the above embodiment.

[0095] In one embodiment, the recovery module is further configured to perform upsampling processing on the compressed non-key frame feature map according to the deconvolution module to obtain the first non-key frame feature map.

[0096] The semantic segmentation module is also used to perform frame-by-frame semantic segmentation on the fused non-keyframe feature map and keyframe feature map based on a fully convolutional multilayer neural network to obtain semantic segmentation results.

[0097] In this embodiment, the description of the recovery module can be found in the description of S410 in the above embodiment. The description of the semantic segmentation module can be found in the description of S710 in the above embodiment.

[0098] In one embodiment, the video semantic segmentation system 100 based on a surveillance scene further includes a training objective construction module and a training module. The training objective construction module is used to construct a target loss function based on the semantic segmentation annotations and semantic segmentation results. The training module is used to train and optimize a fully convolutional multilayer neural network based on the target loss function.

[0099] In this embodiment, the description of the training target construction module can be referred to the description of S80 in the above embodiment. The description of the training module can be referred to the description of S90 in the above embodiment.

[0100] In the various embodiments described above, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to a specific order or hierarchy.

[0101] Those skilled in the art will also understand that the various illustrative logical blocks, modules, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, modules, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0102] The various illustrative logic blocks or modules described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0103] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0104] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for video semantic segmentation based on monitoring a scene, characterized in that, comprises: obtaining a key frame image set and a non-key frame image set; performing feature extraction on each of the key frame images to obtain a key frame feature map; performing resolution compression on each of the non-key frame images to obtain a compressed resolution image, and performing feature extraction on the compressed resolution image to obtain a compressed non-key frame feature map; performing up-sampling processing on the compressed non-key frame feature map to obtain a first non-key frame feature map; performing optical flow mapping transformation on each of the key frame feature maps to obtain a second non-key frame feature map; pixel-by-pixel addition of the first non-key frame feature map and the second non-key frame feature map to obtain a fused non-key frame feature map; performing frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map to obtain a semantic segmentation result; the up-sampling processing on the compressed non-key frame feature map to obtain a first non-key frame feature map, comprising: performing up-sampling processing on the compressed non-key frame feature map according to a deconvolution module to obtain the first non-key frame feature map; the frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map to obtain a semantic segmentation result, comprising: performing frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map according to a fully convolutional multi-layer neural network to obtain the semantic segmentation result.

2. The video semantic segmentation method based on a monitoring scene according to claim 1, wherein the obtaining a key frame image set and a non-key frame image set comprises: obtaining a plurality of video clips, performing semantic segmentation labeling and frame division on each monitoring video clip to form the key frame image set and the non-key frame image set; wherein the number of the non-key frame image set is N times the number of the key frame image set, and N is a positive integer greater than 10. wherein the feature extraction on each of the key frame images to obtain a key frame feature map comprises:

3. The method of claim 2, wherein the video semantic segmentation is based on a monitoring scene, performing feature extraction on each of the key frame images according to a first deep residual network to obtain the key frame feature map; the feature extraction on the compressed resolution image to obtain a compressed non-key frame feature map comprises: performing feature extraction on the compressed resolution image according to a second deep residual network to obtain the compressed non-key frame feature map; wherein the network structure of the backbone network of the first deep residual network is greater than the network structure of the backbone network of the second deep residual network. After the frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map according to a fully convolutional multi-layer neural network to obtain the semantic segmentation result, the method further comprises: constructing a target loss function according to the semantic segmentation labeling and the semantic segmentation result; 4.The method of claim 1, wherein, training and optimizing the fully convolutional multi-layer neural network according to the target loss function. comprises: a data acquisition module for obtaining a key frame image set and a non-key frame image set; 5.A video semantic segmentation system based on monitoring a scene, characterized in that, a key frame feature extraction module for performing feature extraction on each of the key frame images to obtain a key frame feature map; ​ ​ a compression feature extraction module, configured to perform resolution compression on each non-key frame image to obtain a compressed resolution image, and perform feature extraction on the compressed resolution image to obtain a compressed non-key frame feature map; a recovery module, configured to perform up-sampling processing on the compressed non-key frame feature map to obtain a first non-key frame feature map; a mapping module, configured to perform optical flow mapping transformation on each key frame feature map to obtain a second non-key frame feature map; a fusion module, configured to perform pixel-by-pixel addition on the first non-key frame feature map and the second non-key frame feature map to obtain a fused non-key frame feature map; a semantic segmentation module, configured to perform frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map to obtain a semantic segmentation result; the recovery module is further configured to perform up-sampling processing on the compressed non-key frame feature map according to a deconvolution module to obtain the first non-key frame feature map; the semantic segmentation module is further configured to perform frame-by-frame semantic segmentation on the fused non-key frame feature map and the key frame feature map according to a full convolution multi-layer neural network to obtain the semantic segmentation result. 6.The video semantic segmentation system based on a monitoring scene according to claim 5, characterized in that, The data acquisition module is configured to acquire a plurality of video clips, perform semantic segmentation labeling and frame division on each monitoring video clip to form the key frame image set and the non-key frame image set. The number of the non-key frame image set is N times the number of the key frame image set, and N is a positive integer greater than 10. 7.The video semantic segmentation system based on a monitoring scene according to claim 6, characterized in that, The key frame feature extraction module is further configured to perform feature extraction on each key frame image according to a first deep residual network to obtain the key frame feature map. The compression feature extraction module is further configured to perform feature extraction on the compressed resolution image according to a second deep residual network to obtain the compressed non-key frame feature map. The network structure of the backbone network of the first deep residual network is greater than the network structure of the backbone network of the second deep residual network. 8.The video semantic segmentation system based on a monitoring scene according to claim 5, characterized in that, The system further comprises: a training target construction module, configured to construct a target loss function according to the semantic segmentation labeling and the semantic segmentation result; a training module, configured to train and optimize the full convolution multi-layer neural network according to the target loss function.

Images