Method and system to provide video semantic segmentation, and method to estimate occlusion regions

Abstract

This invention provides a system and method for providing video semantic segmentation. Semantic segmentation is performed on a first frame in a sequence of video frames to obtain at least one first semantic feature of the first frame. Semantic segmentation is performed on a second frame in the sequence to obtain at least one second semantic feature of the second frame, wherein the second frame follows the first frame. Semantic segmentation is performed on a third frame in the sequence to obtain at least one third semantic feature, wherein the third frame follows and is also following the first frame by a first predetermined number of consecutive frames. At least one first semantic feature, at least one second semantic feature, and at least one third semantic feature are combined to form at least one fourth semantic feature of the second frame.

Description

Technical Field

[0001] The subject matter disclosed in this paper relates to video semantic segmentation. More specifically, the subject matter disclosed in this paper relates to a video semantic segmentation system and method that uses bidirectional feature propagation based on learned occlusion mapping and attention-based feature correction. Cross-reference of related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 254,051, filed October 8, 2021, the disclosure of which is incorporated herein by reference in its entirety. Prior Technology

[0003] Video semantic segmentation is a typical computer vision problem involving the transformation of input video into a sequence of corresponding masks with highlighted and classified regions of interest. More specifically, video semantic segmentation is the process of classifying each pixel in each frame of a video into belonging to a specific object class. Due to high computational costs, per-frame video semantic segmentation may not be an acceptable approach. Besides per-frame segmentation accuracy, temporal coherence is another key aspect of video semantic segmentation that per-frame image semantic segmentation methods cannot optimally handle. To achieve good performance in both segmentation accuracy and temporal coherence, the process can utilize temporal information existing in adjacent frames when performing segmentation on a given frame in the video. Content continuity within consecutive frames can be advantageously used for stream-based feature propagation; however, the results can be distorted due to inaccuracies in the estimated optical flow. Summary of the Invention

[0004] An example embodiment provides a method for providing video semantic segmentation, wherein the method may include: performing semantic segmentation on a first frame in a video frame sequence by a first video semantic segmentation network to obtain at least one first semantic feature of the first frame; performing semantic segmentation on a second frame in the video frame sequence by a second video semantic segmentation network to obtain at least one second semantic feature of the second frame, the second frame following the first frame; performing semantic segmentation on a third frame in the video frame sequence by a third video semantic segmentation network to obtain at least one third semantic feature of the third frame, the third frame following both the first and second frames according to a predetermined number of consecutive frames; and generating at least one fourth semantic feature of the second frame by a combining network based on at least one first semantic feature, at least one second semantic feature, and at least one third semantic feature. In one embodiment, a predetermined number of consecutive frames may comprise a fixed number of consecutive frames, with a first frame at the beginning of the fixed number of consecutive frames, a third frame at the end of the fixed number of consecutive frames, and a second frame being a single frame in the middle of the fixed number of consecutive frames between the first and third frames. The first and third frames may be key frames. In another embodiment, the first image semantic segmentation network may be a deeper neural network than the second image semantic segmentation network, and the third image semantic segmentation network may be a deeper neural network than the second image semantic segmentation network. In yet another embodiment, the method may further include: propagating at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; and propagating at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature. In yet another embodiment, the method may further include: determining, by a first optical flow network, at least one first forward optical flow from a fourth frame to a second frame in a video frame sequence and at least one first backward optical flow from a second frame to a fourth frame, the fourth frame being between the first frame and the second frame, and the fourth frame being immediately adjacent to the second frame; and determining, by a second optical flow network, at least one second forward optical flow from a second frame to a fifth frame in a video frame sequence and at least one second backward optical flow from a fifth frame to a second frame, the fifth frame being between the second frame and the third frame, and the fifth frame being immediately adjacent to the second frame.In one embodiment, the method may further include: propagating at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; propagating at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature; determining at least one first occlusion mapping from the fourth frame to the second frame and at least one second occlusion mapping from the fifth frame to the second frame by the first occlusion network at least in part based on at least one first forward optical flow, at least one first backward optical flow, at least one second forward optical flow, and at least one second backward optical flow; and generating at least one third occlusion mapping for the second frame based on at least one first occlusion mapping and at least one second occlusion mapping. In another embodiment, the method may further include: generating a semantic segmentation frame of a second frame by a combination network based on at least one first propagation semantic feature and at least one first masking map, at least one second propagation semantic feature and at least one second masking map, and at least one second semantic feature and at least one third masking map of the second frame by: multiplying at least one first propagation semantic feature with at least one first masking map element-wise to obtain at least one first weighted semantic feature; multiplying at least one second propagation semantic feature with at least one second masking map element-wise to obtain at least one second weighted semantic feature; multiplying at least one second semantic feature with at least one third masking map element-wise to obtain at least one third weighted semantic feature; and multiplying at least one first weighted semantic feature, at least one second weighted semantic feature, and at least one third weighted semantic feature element-wise.

[0005] An example embodiment provides a method for estimating occlusion regions in a sequence of video frames. The method may include: determining, by a first optical flow network, at least one first forward optical flow from a first frame to a second frame and at least one first backward optical flow from a second frame to a first frame, wherein the first frame precedes the second frame in time and the second frame immediately follows the first frame; and determining, by a second optical flow network, at least one second forward optical flow from the second frame to a third frame and at least one second backward optical flow from the third frame to the second frame, wherein the second frame precedes the third frame in time and the third frame immediately follows the second frame. In one embodiment, the method may further include: determining at least one first occlusion mapping from a first frame to a second frame and at least one second occlusion mapping from a third frame to a second frame by a first occlusion network at least in part based on at least one first forward optical flow, at least one first backward optical flow, at least one second forward optical flow, and at least one second backward optical flow; and generating at least one third occlusion mapping for the second frame by a combining network based on at least one first occlusion mapping and at least one second occlusion mapping. In another embodiment, the method may further include: performing semantic segmentation on a fourth frame in a video frame sequence using a first image semantic segmentation network to obtain at least one first semantic feature of the fourth frame, the fourth frame being temporally preceding the second frame; performing semantic segmentation on the second frame using a second image semantic segmentation network to obtain at least one second semantic feature of the second frame, the second frame being following the fourth frame; performing semantic segmentation on a fifth frame in a video frame sequence using a third image semantic segmentation network to obtain at least one third semantic feature of the fifth frame, the fifth frame being following the second frame and also following the fourth frame via a first predetermined number of consecutive frames; and generating at least one fourth semantic feature of the second frame using a combining network based on at least one first semantic feature, at least one second semantic feature, and at least one third semantic feature. In yet another embodiment, the first predetermined number of consecutive frames may comprise a fixed number of consecutive frames, the fourth frame may be at the beginning of the fixed number of consecutive frames, the fifth frame may be at the end of the fixed number of consecutive frames, the second frame may be a single frame between the fourth and fifth frames in the middle of the fixed number of consecutive frames, and the fourth and fifth frames may be key frames. In yet another embodiment, the first image semantic segmentation network may be a deeper neural network than the second image semantic segmentation network, and the third image semantic segmentation network may be a deeper neural network than the second image semantic segmentation network. In one embodiment, the method may further include: propagating at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; and propagating at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature.In another embodiment, the method may further include: generating a semantic segmentation frame of a second frame by a combination network based on at least one first propagation semantic feature and at least one first masking map, at least one second propagation semantic feature and at least one second masking map, and at least one second semantic feature and at least one third masking map of the second frame by: multiplying at least one first propagation semantic feature with at least one first masking map element-wise to obtain at least one first weighted semantic feature; multiplying at least one second propagation semantic feature with at least one second masking map element-wise to obtain at least one second weighted semantic feature; multiplying at least one second semantic feature with at least one third masking map element-wise to obtain at least one third weighted semantic feature; and multiplying at least one first weighted semantic feature, at least one second weighted semantic feature, and at least one third weighted semantic feature element-wise.

[0006] An example embodiment provides a system for providing video semantic segmentation. The system may include a first semantic segmentation network, a second semantic segmentation network, a third semantic segmentation network, and a combination network. The first semantic segmentation network can be configured to perform semantic segmentation on a first frame within a predetermined number of consecutive video frames to obtain at least one first semantic feature of the first frame. The second video semantic segmentation network can be configured to perform semantic segmentation on a second frame in a sequence of video frames to obtain at least one second semantic feature of the second frame, the second frame following the first frame. The third semantic segmentation network can be configured to perform semantic segmentation on a third frame in a sequence of video frames to obtain at least one third semantic feature of the third frame, the third frame following and also following the first frame within a first predetermined number of consecutive frames. The combination network can be configured to generate at least one fourth semantic feature of the second frame based on at least one first semantic feature, at least one second semantic feature, and at least one third semantic feature. In one embodiment, a first predetermined number of consecutive frames may comprise a fixed number of consecutive frames, with the first frame at the beginning of the fixed number of consecutive frames, the third frame at the end of the fixed number of consecutive frames, and the second frame being a single frame in the middle of the fixed number of consecutive frames between the first and third frames. The first and third frames may be key frames. In another embodiment, the system may be further configured to: propagate at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; and propagate at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature. In yet another embodiment, the system may further include: a first optical flow network configured to determine at least one first forward optical flow from a fourth frame to a second frame and at least one first backward optical flow from a second frame to a fourth frame, the fourth frame being between the first frame and the second frame, and the fourth frame being immediately adjacent to the second frame; and a second optical flow network configured to determine at least one second forward optical flow from a second frame to a fifth frame and at least one second backward optical flow from a fifth frame to a second frame, the fifth frame being between the second frame and the third frame, and the fifth frame being immediately adjacent to the second frame. In yet another embodiment, the system may further include: a first masking network configured to determine at least one first masking map from a fourth frame to a second frame and at least one second masking map from a fifth frame to a second frame based at least in part on at least one first forward optical flow, at least one first backward optical flow, at least one second forward optical flow, and at least one second backward optical flow; and the combined network may be further configured to generate at least one third masking map for the second frame based on at least one first masking map and at least one second masking map.In one embodiment, the system may be further configured to: propagate at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; and propagate at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature, wherein the combined network is further configured to generate based on at least one first propagated semantic feature and at least one first masking map, at least one second propagated semantic feature and at least one second masking map, and at least one second semantic feature of the second frame and at least one third masking map of the second frame by the following operations. The semantic segmentation of the second frame involves: multiplying at least one first propagation semantic feature with at least one first masking mapping element-wise to obtain at least one first weighted semantic feature; multiplying at least one second propagation semantic feature with at least one second masking mapping element-wise to obtain at least one second weighted semantic feature; multiplying at least one second semantic feature with at least one third masking mapping element-wise to obtain at least one third weighted semantic feature; and multiplying at least one first weighted semantic feature, at least one second weighted semantic feature, and at least one third weighted semantic feature element-wise. Simple Explanation of the Diagram

[0007] In the following sections, the nature of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the figures: Figure 1 depicts an example embodiment of a system framework based on the subject matter disclosed herein. Figure 2 depicts the input frame and the output optical flow generated by FlowNet according to the subject matter disclosed herein. Figure 3A depicts the input of four estimated optical flows into the shading network OccNet to estimate regions that may contain distortions caused by shading during the propagation process according to the subject disclosed herein. Figure 3B depicts an example U-Net shape structure based on the subject matter disclosed herein, which can be used to receive four optical flows from three consecutive frames as input and output an estimate of the occlusion mapping. Figure 4 depicts the training configuration of the semantic segmentation propagation component of the framework revealed in this paper. Figure 5 illustrates an example visual intermediate result of an example embodiment of the bidirectional feature propagation framework based on the subject matter disclosed herein. Figure 6 illustrates the results of instance visual semantic segmentation of five sequences of three non-key visual frames based on the topics disclosed in this paper. Figure 7 illustrates a comparison of the visual results of different semantic segmentation techniques compared to the semantic segmentation techniques with bidirectional feature propagation and attention-based feature correction techniques disclosed in this paper. Figure 8 depicts an electronic device, which in one embodiment may include a video semantic segmentation system using bidirectional feature propagation and attention-based feature correction in accordance with the topics disclosed herein. Implementation

[0008] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of this disclosure. However, those skilled in the art will understand that the disclosed forms can be practiced without such specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the subject matter disclosed herein.

[0009] Throughout this specification, the reference to "an embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in an embodiment," "according to an embodiment," or "according to an embodiment" (or other phrases with similar meanings) appearing in various places throughout this specification may not necessarily all refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the term "illustrative" means "serving as an example, example, or illustration." Any embodiment described herein as "illustrative" is not necessarily to be construed as preferred or superior to other embodiments. Additionally, a particular feature, structure, or characteristic may be combined in any suitable manner in one or more embodiments. Moreover, depending on the context of the discussion herein, singular terms may contain corresponding plural forms, and plural terms may contain corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "pre-defined", "pixel-specific", etc.) may be used interchangeably with their non-hyphenated counterparts (e.g., "two-dimensional", "pre-defined", "pixel-specific", etc.), and uppercase terms (e.g., "counter clock", "column select", "PIXOUT", etc.) may be used interchangeably with their non-uppercase counterparts (e.g., "counter clock", "column select", "pixout", etc.). Such occasional interchangeability should not be considered inconsistent with each other.

[0010] Furthermore, depending on the context of the discussion herein, singular terms may contain corresponding plural forms, and plural terms may contain corresponding singular forms. It should be further noted that the various diagrams (including component diagrams) illustrated and discussed herein are for illustrative purposes only and are not drawn to scale. For example, for clarity, the dimensions of some components may be enlarged relative to other components. Additionally, reference numerals have been repeated in the diagrams where deemed appropriate to indicate corresponding and / or similar components.

[0011] The terminology used herein is for the purpose of describing some exemplary embodiments only and is not intended to limit the subject matter claimed. As used herein, unless the context clearly indicates otherwise, the singular forms "a / an" and "described" are intended to include the plural forms as well. It should be further understood that the term "comprise / comprising," when used in this specification, designates the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the terms "first," "second," etc., are used as labels for their preceding nouns and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless so explicitly defined. Furthermore, the same schematic element symbols can be used across two or more schematics to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality. However, this usage is for illustrative and explanatory purposes only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0012] It should be understood that when an element or layer is referred to as being on, "connected to," or "coupled to" another element or layer, the element or layer may be directly on, directly connected to, or directly coupled to the other element or layer, or there may be intervening elements or layers. Conversely, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers. Similar numbers always refer to similar elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0013] As used herein, the terms "first," "second," etc., serve as labels for their preceding nouns and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same schematic element symbols may be used across two or more schematics to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality. However, this usage is merely for simplification and ease of discussion; it does not imply that the construction or architectural details of such components or units are identical across all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0014] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary knowledge in the field to which this subject belongs. Furthermore, it should be understood that terms such as those defined in common dictionaries shall be interpreted as having the same meaning in the context of the relevant art, and shall not be interpreted in an idealized or overly formal sense, unless expressly defined herein.

[0015] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with a module. For example, software may be embodied as a software suite, code, and / or instruction set or instructions, and the term "hardware" as used in any implementation described herein may individually or in any combination include, for example, assemblies, wired circuit systems, programmable circuit systems, state machine circuit systems, and / or firmware storing instructions executable by programmable circuitry. Modules may collectively or individually be embodied as circuit systems forming part of a larger system, such as, but not limited to, integrated circuits (ICs), system-on-chips (SoCs), assemblies, etc.

[0016] The subject matter disclosed herein provides a system and method for achieving high segmentation accuracy and temporal consistency in video semantic segmentation with low computational cost by using bidirectional feature propagation and attention-based feature correction. The bidirectional feature propagation method propagates features of key frames towards non-key frames between key frames in both the forward and backward temporal directions. The attention-based feature correction method uses features obtained from the current frame by a lightweight network based on occlusion mappings learned by an occlusion network taking into account occlusion detection in both the forward and backward directions to improve the propagated features.

[0017] Several traditional keyframe-based video semantic segmentation techniques utilize content continuity between frames and propagate high-order features extracted at key frames to other temporal steps in one direction. This traditional propagation process can include misalignment caused by motion between frames. For example, Deep Feature Flow (DFF) uses optical flow to warp high-order features from key frames to the next frame. A correction fusion network named Accel uses low-order features extracted from the current frame to update the warped features. Compared to traditional methods, the subject matter disclosed in this paper uses content continuity between frames to estimate the semantic segmentation of non-key frames in both the forward and backward temporal directions, and uses lightweight networks to improve distortion, thereby providing a high-quality segmentation mapping sequence with low computational cost.

[0018] Figure 1 depicts an example embodiment of a system framework 100 based on the subject matter disclosed herein. System framework 100 includes a bidirectional feature propagation framework and an attention-based feature correction mechanism. The bidirectional feature propagation framework propagates features from critical frames to non-critical frames in both the forward and backward directions in time. The attention-based feature correction mechanism fuses the forward and backward propagated features, as well as features extracted from the current frame based on an occlusion mapping learned by the occlusion network taking into account occlusion detection in both the forward and backward directions.

[0019] Referring to Figure 1, the bidirectional feature propagation framework 100 includes a first image semantic segmentation network (SegNetk) 101, a second image semantic segmentation network (SegNetk+D) 102, a third image semantic segmentation network (SegNetnk) 103, a first optical flow network (FlowNet) 104, a second optical flow network (FlowNet) 105, a first warping operation (W) circuit 106, and a second warping operation (W) circuit 107. The attention-based feature correction mechanism includes a masking network (OccNet) 108. The outputs of the bidirectional feature propagation framework and the masking network are combined in a combining network 109. The various components forming the bidirectional feature propagation framework 100 can be implemented as one or more modules, networks, circuits, and / or discrete components.

[0020] Each video frame in the sequence of video frames depicted at the top of Figure 1 can be considered a key frame or a non-key frame. Key frames can be directly applied to the image semantic segmentation network SegNetk 101 or the image semantic segmentation network SegNetk+D 102 to obtain semantic segmentation results (i.e., semantic features). The features of the key frames can then be propagated frame-by-frame to temporally subsequent or temporally preceding non-key frames (i.e.,... ).

[0021] Optical flow can be used as a guide and bilinear interpolation can be used as a warping operator. That is, the predicted optical flow pairs of the images output from FlowNet 104 and FlowNet 105, such as... and It can be used to propagate features towards subsequent and preceding non-critical frames, respectively. For non-critical frames... There are four estimated optical flows ,in and The forward and backward flows between a pair of images are indicated respectively. Figure 2 depicts the input frame and the output optical flow generated by FlowNet 104 and FlowNet 105.

[0022] As depicted in Figure 3A, the four estimated optical flows can also be input into the OccNet 108 shading network to estimate regions that may contain distortions caused by shading during propagation. The predicted shading can then be mapped... and occlusion mapping Guided by the propagation features, the correction is performed to correct the errors caused by using features extracted from the current frame by SegNetnk 103. Any remaining slight distortion is indicated; SegNetnk 103 is a relatively lightweight image segmentation network compared to SegNetk 101 and SegNetk 102.

[0023] As is traditionally performed, the bidirectional feature propagation method disclosed in this paper utilizes two keyframes instead of a single keyframe. It assumes a distance between the two keyframes... If fixed, the key frames can be set separately. [ I k ] and key frames [ I k+D The input is fed into the image semantic segmentation network SegNetk 101 and the image semantic segmentation network SegNetk+D 102 to obtain the corresponding semantic segmentation features, respectively. and semantic segmentation features For the forward direction, optical flow can be calculated or determined. And using the optical flow to Propagate to relative to the key frame [ I k The subsequent frames in time. For the backward direction, optical flow can be determined. And using the optical flow to Propagate to relative to the key frame [I k+D [Previous message frame in time]

[0024] Supervised learning can be used to train the bidirectional semantic segmentation component of Frame 100. As depicted in Figure 4, a batch of three images can be segmented... The input is fed into the semantic segmentation and propagation part of frame 100. (Image) Includes live footage. An index of images within a batch can be generated based on the following equation: [,] [](1) [ ] and (2)

[0025] During the bidirectional semantic segmentation portion of training framework 100, the framework may contain two main semantic segmentation networks, namely, an image segmentation network. Network 201 (corresponding to SegNetk and SegNetk+D networks in Figure 1) and FlowNet 102 (corresponding to FlowNet 104 and FlowNet 105 in Figure 1). It can be pre-trained on, for example, ImageNet, and then fine-tuned on a specific segmentation dataset (e.g., Cityscapes). CNNStream202 can be pre-trained on, for example, the Flying Chairs dataset, and then fine-tuned during the training of the bidirectional frame, while the network... Keep it fixed.

[0026] The network will and The semantic segmentation output is and From The output of the flow network 202 is determined by the warp operator. The operations are performed and combined to form the estimated semantic segmentation. (Compared with the actual GT).

[0027] An attention-based feature correction mechanism is used to fuse features from the forward and backward directions with features extracted from the current frame based on an occlusion mapping learned by the occlusion network considering occlusion detection in both the forward and backward directions. In some embodiments, as depicted in FIG3B, the occlusion network may, for example, have a U-Net shape structure 350, which incorporates features from three consecutive frames. The four optical flow views are taken as input and the occlusion mapping is estimated. and occlusion mapping In Figure 3B, the term "Conv2d" refers to a two-dimensional (2D) convolutional layer, "Concat" refers to a cascaded layer, and "Deconv" refers to a deconvolutional layer. Furthermore, the numbers "8," "16," "32," and "64" in Figure 3B indicate the number of channels following each convolutional layer. Based on complementary optical flow, this can be achieved within the frame... The area in the frame The forward direction that is obscured Above and in the frame The area in the frame The obscured rearward direction Predictive occlusion mapping. Regions where forward and backward optical flow are ambiguous for classification can be ignored. From the current frame. The guidance mapping can correct errors caused by features propagating in the forward or backward direction.

[0028] Referring back to Figure 1, the instance framework 100 may include a first image semantic segmentation network SegNetk 101, a second image semantic segmentation network SegNetk+D 102, a third image semantic segmentation network SegNetnk 103, a first optical FlowNet 104, a second optical FlowNet 105, a first warping operation W circuit 106, a second warping operation W circuit 107, an occlusion network OccNet 108, and a combination network 109. In one embodiment, SegNetnk may include ten convolutional layers interleaved with batch normalization and leaky rectified linear unit (ReLU) layers for feature encoding, and four deconvolutional layers interleaved with leaky ReLU layers for feature decoding. The SegNetnk network can be trained separately for a specific segmentation dataset (e.g., Cityscapes). Next, OccNet 108 can be trained with arbitrary initialized weights, while SegNetk, SegNetk+D, FlowNet networks, and SegNetnk will remain fixed.

[0029] make and These represent the propagation characteristics from the key frame to the current frame, and make... To utilize the features extracted from the current frame using SegNetnk, the combined network 109 can employ weighted summation to perform feature rectification. Therefore: (3) in" "" indicates element-wise multiplication of space.

[0030] In one example embodiment of framework 100, the DeepLabv3+ network can be used as a semantic segmentation architecture for SegNetk and SegNetk+D networks based on its performance in terms of accuracy and efficiency. In another example embodiment, a modified FlowNet2-S can be used as a CNN of the optical flow estimation network FlowNet.

[0031] The bidirectional feature propagation framework 100 disclosed in this paper can be used to evaluate datasets such as Cityscapes, a popular dataset in the fields of semantic segmentation and automated driving. The training set and validation set contain 2975 and 500 video segments, respectively. Each video segment has 30 frames, with the 20th frame annotated by pixel-level semantic labels with 19 categories.

[0032] Figure 5 illustrates an example visual intermediate result of an embodiment of the bidirectional feature propagation framework 100, including both forward and backward directions. and Features propagated from the current frame, features extracted from the current frame. ) and occlusion mapping ( , as well as It can be observed that the masking network disclosed in this paper can estimate the masking mapping in both the forward and backward directions, and can highlight two areas where the propagation features are unclear for prediction and can be used to improve based on the features of the current frame.

[0033] Figure 6 illustrates the results of five sequences of example visual semantic segmentation of three (3) non-critical visual frames based on the topics disclosed herein. The frame index is indicated at the top of Figure 6. The top column of Figure 6 shows the example input frames. The second column shows the forward propagation results. The third column shows the backward propagation results. The fourth column shows the current results, and the fifth column shows the final output.

[0034] The performance of different video semantic segmentation methods can be evaluated by measuring segmentation accuracy using Mean Over Union (mIoU) and mIoU-based temporal consistency (mTC). Table 1 illustrates the performance of example embodiments of the bidirectional feature propagation framework disclosed herein compared to traditional keyframe-based techniques including DFF, Accel, and Distortion-Aware Video Semantic Segmentation (DAVSS). In Table 1, the bidirectional propagation framework disclosed herein is presented in the bottom two columns. At the top of the bottom two columns, the bidirectional propagation framework uses DeeplabV3+ networks as SegNetk and SegNetk+D, and at the bottom of the two columns, the bidirectional propagation framework uses HRNetV2 networks as SegNetk and SegNetk+D. For both example embodiments, the propagation distance D is set to 5. It can be observed that the bidirectional propagation framework revealed in this paper outperforms other methods in terms of mTC and has an mIoU value comparable to the favorable computational burden in GFLOP. Table 1. Evaluation of the Cityscapes and CamVid datasets. Citiscapes CamVid method mIoU mTC GFLOP mIoU mTC GFLOP DeeplabV3+ 76.6 76.6 820 72.0 83.2 270 HRNetV2 75.9 81.0 156 75.0 83.9 52 GRFP 76.6 83.8 468 74.6 87.2 156 TDNet 76.5 81.6 161 72.6 84.7 54 DFF 68.7 - 180 66.0 - 60 Accel 72.1 - 510 66.7 - 170 DAVSS 75.4 84.5 212 71.1 85.0 72 BIDIR(DeeplabV3+) 76.5 83.9 231 71.8 84.1 78 BIDIR(HRNetV2) 75.7 86.5 231 74.4 88.4 78

[0035] Figure 7 illustrates a comparison of instance visual results between different semantic segmentation techniques and the semantic segmentation technique with bidirectional feature propagation and attention-based feature correction disclosed in this paper. The top column of Figure 7 shows four instance frames. The second column shows the results for each individual instance frame provided by the DeeplabV3+ network. The third column shows the results provided by the DAVSS network. The fourth column shows the results provided by the semantic segmentation technique with bidirectional feature propagation and attention-based feature correction disclosed in this paper. The fifth column shows the actual results of the four instance frames.

[0036] Figure 8 depicts an electronic device 800. In one embodiment, the electronic device 800 may include a video semantic segmentation system using bidirectional feature propagation and attention-based feature correction, according to the subject matter disclosed herein. The electronic device 800 and its various system components may be formed from one or more modules. The electronic device 800 may include a controller (or CPU) 810, input / output devices 820 (such as, but not limited to, a keypad, keyboard, display, touchscreen display, 2D image sensor, 3D image sensor), memory 830, interface 840, GPU 850, imaging processing unit 860, neural processing unit 870, and TOF processing unit 880, all coupled to each other via bus 890. The controller 810 may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, or the like. The memory 830 may be configured to store command codes to be used by the controller 810 and / or to store user data.

[0037] In one embodiment, the imaging processing unit 860 may include a video semantic segmentation system using bidirectional feature propagation and attention-based feature correction, according to the topics disclosed herein. In another embodiment, the neural processing unit 870 may be part of a video semantic segmentation system using bidirectional feature propagation and attention-based feature correction, according to the topics disclosed herein.

[0038] Interface 840 may be configured to include a wireless interface, which is configured to transmit data to, for example, a wireless communication network or receive data from a wireless communication network using RF signals. Wireless interface 840 may also include, for example, an antenna. Electronic System 800 can also be used in communication interface protocols of communication systems, such as, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E-TDMA), Wideband CDMA (WCDMA), CDMA2000, Wi-Fi, Municipal Wi-Fi (Muni Wi-Fi), Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast low-latency access with seamless handoff Orthogonal Frequency Division Multiplexing (Flash-OFDM), and IEEE 802.20. General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, WiMAX Advanced, Universal Mobile Telecommunication Service-Time Division Duplex (UMTS-TDD), High Speed ​​Packet Access (HSPA), Evolution Data Optimized (EVDO), Long Term Evolution-Advanced (LTE-Advanced), Multichannel Multipoint Distribution Service (MMDS), Fifth-Generation Wireless (5G), Sixth-Generation Wireless (6G), etc.

[0039] The embodiments of the subject matter and operation described in this specification may be implemented as digital electronic circuit systems or as computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents) or a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagated signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be contained in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, when the computer storage medium is not a propagated signal, the computer storage medium may be a source or destination of computer program instructions encoded in artificially generated propagated signals. Computer storage media may also be one or more individual physical components or media (e.g., multiple CDs, magnetic disks, or other storage devices) or contained within one or more individual physical components or media. Furthermore, the operations described in this specification can be performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0040] While this specification may contain numerous specific implementation details, these details should not be construed as limiting the scope of any claimed subject matter, but rather as descriptions of features specific to particular embodiments. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and are initially even claimed in this manner, one or more features from the claimed combination may be removed from that combination in some situations, and the claimed combination may be for sub-combinations or variations thereof.

[0041] Similarly, although operations are depicted in a specific order in the diagrams, this should not be construed as requiring the execution of such operations in the specific order shown or in sequential order, or performing all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0042] Therefore, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0043] As will be appreciated by those skilled in the art, the novel concepts described herein can be modified and altered across a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any of the specific illustrative teachings discussed above, but is actually defined by the following scope of the patent application.

[0044] 100: System Framework 101: First-level image semantic segmentation network 102: Second Image Semantic Segmentation Network 103: Third-generation image semantic segmentation network 104: First Optical Flow Network 105: Second Optical Flow Network 106: First warp operation circuit 107: Second Warp Operation Circuit 108: Covering the network 109: Combined Network 201: Image Segmentation Network 202: Convolutional Neural Networks Flow Networks 350: U-Net shape structure 800: Electronic Devices 810: Controller 820: Input / Output Device 830: Memory 840: Interface 850: GPU 860: Imaging Processing Unit 870: Neural Processing Unit 880: TOF Processing Unit 890: Busbar Current message box Estimated optical flow :optical flow GT: Pro Evolution Soccer :feature Semantic segmentation features :image Non-critical frames , :Mask mapping , Semantic Segmentation : Estimated semantic segmentation

Claims

1. A method for providing video semantic segmentation, the method comprising: The first frame in the video frame sequence is semantically segmented by the first image semantic segmentation network to obtain at least one first semantic feature of the first frame; A second image semantic segmentation network performs semantic segmentation on a second frame in the video frame sequence to obtain at least one second semantic feature of the second frame, which follows the first frame; a third image semantic segmentation network performs semantic segmentation on a third frame in the video frame sequence to obtain at least one third semantic feature of the third frame, which follows both the first and second frames according to a predetermined number of consecutive frames; And at least one fourth semantic feature generated by the combined network based on the at least one first semantic feature, the at least one second semantic feature, and the at least one third semantic feature, wherein the predetermined number of consecutive frames includes a fixed number of consecutive frames, wherein the first frame is at the beginning of the fixed number of consecutive frames, wherein the third frame is at the end of the fixed number of consecutive frames, wherein the second frame includes a single frame between the first frame and the third frame in the middle of the fixed number of consecutive frames, and wherein the first frame and the third frame include key frames.

2. The method of claim 1, wherein the first image semantic segmentation network includes a deeper neural network than the second image semantic segmentation network, and wherein the third image semantic segmentation network includes a deeper neural network than the second image semantic segmentation network.

3. The method as described in claim 1, further comprising: At least one first semantic feature is propagated toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; And propagate the at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature.

4. The method as described in claim 1, further comprising: The first optical flow network determines at least one first forward optical flow from the fourth frame to the second frame of the video frame sequence and at least one first backward optical flow from the second frame to the fourth frame, wherein the fourth frame is between the first frame and the second frame, and the fourth frame is immediately adjacent to the second frame; The second optical flow network determines at least one second forward optical flow from the second frame to the fifth frame of the video frame sequence and at least one second backward optical flow from the fifth frame to the second frame, wherein the fifth frame is between the second frame and the third frame, and the fifth frame is immediately adjacent to the second frame.

5. The method as described in claim 4, further comprising: At least one first semantic feature is propagated toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; At least one third semantic feature is propagated toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature; The first masking network determines at least one first masking map from the fourth frame to the second frame and at least one second masking map from the fifth frame to the second frame, based at least in part on the at least one first forward optical flow, the at least one first backward optical flow, the at least one second forward optical flow, and the at least one second backward optical flow; and generates at least one third masking map for the second frame based on the at least one first masking map and the at least one second masking map.

6. The method of claim 5, further comprising generating a semantic segmentation frame of the second frame by the combined network based on the at least one first propagation semantic feature and the at least one first masking map, the at least one second propagation semantic feature and the at least one second masking map, and the at least one second semantic feature of the second frame and the at least one third masking map of the second frame by: multiplying the at least one first propagation semantic feature with the at least one first masking map by element-wise multiplication to obtain at least one first weighted semantic feature; multiplying the at least one second propagation semantic feature with the at least one second masking map by element-wise multiplication to obtain at least one second weighted semantic feature; multiplying the at least one second semantic feature with the at least one third masking map by element-wise multiplication to obtain at least one third weighted semantic feature; and multiplying the at least one first weighted semantic feature, the at least one second weighted semantic feature, and the at least one third weighted semantic feature by element-wise summation.

7. A method for estimating occlusion regions in a sequence of video frames, the method comprising: A first optical flow network determines at least one first forward optical flow from a first frame to a second frame and at least one first backward optical flow from the second frame to the first frame, wherein the first frame is time-preceding the second frame and the second frame is immediately following the first frame; a second optical flow network determines at least one second forward optical flow from the second frame to a third frame and at least one second backward optical flow from the third frame to the second frame, wherein the second frame is time-preceding the third frame and the third frame is immediately following the second frame; The first masking network determines at least one first masking map from the first frame to the second frame and at least one second masking map from the third frame to the second frame based at least in part on the at least one first forward optical flow, the at least one first backward optical flow, the at least one second forward optical flow, and the at least one second backward optical flow; and the combining network generates at least one third masking map for the second frame based on the at least one first masking map and the at least one second masking map.

8. The method as described in claim 7, further comprising: The fourth frame in the video frame sequence is semantically segmented by a first image semantic segmentation network to obtain at least one first semantic feature of the fourth frame, which is temporally preceding the second frame; the second frame is semantically segmented by a second image semantic segmentation network to obtain at least one second semantic feature of the second frame, which is following the fourth frame; the fifth frame is semantically segmented by a third image semantic segmentation network to obtain at least one third semantic feature of the fifth frame, which is following the second frame and also following the fourth frame by a first predetermined number of consecutive frames; and the combining network generates at least one fourth semantic feature of the second frame based on the at least one first semantic feature, the at least one second semantic feature, and the at least one third semantic feature.

9. The method of claim 8, wherein the first predetermined number of consecutive frames comprises a fixed number of consecutive frames, and wherein the fourth frame is at the beginning of the fixed number of consecutive frames, wherein the fifth frame is at the end of the fixed number of consecutive frames, wherein the second frame comprises a single frame between the fourth frame and the fifth frame in the middle of the fixed number of consecutive frames, and wherein the fourth frame and the fifth frame comprise key frames.

10. The method of claim 8, wherein the first image semantic segmentation network includes a deeper neural network than the second image semantic segmentation network, and wherein the third image semantic segmentation network includes a deeper neural network than the second image semantic segmentation network.

11. The method as described in claim 8, further comprising: At least one first semantic feature is propagated toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; And propagate the at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature.

12. The method of claim 11, further comprising generating a semantic segmentation frame of the second frame by the combined network based on the at least one first propagation semantic feature and the at least one first masking map, the at least one second propagation semantic feature and the at least one second masking map, and the at least one second semantic feature of the second frame and the at least one third masking map of the second frame by: multiplying the at least one first propagation semantic feature with the at least one first masking map by element-wise multiplication to obtain at least one first weighted semantic feature; multiplying the at least one second propagation semantic feature with the at least one second masking map by element-wise multiplication to obtain at least one second weighted semantic feature; multiplying the at least one second semantic feature with the at least one third masking map by element-wise multiplication to obtain at least one third weighted semantic feature; and multiplying the at least one first weighted semantic feature, the at least one second weighted semantic feature, and the at least one third weighted semantic feature by element-wise summation.

13. A system for providing video semantic segmentation, the system comprising: The first semantic segmentation network is configured to perform semantic segmentation on a first frame in a predetermined number of consecutive video frames to obtain at least one first semantic feature of the first frame. A second semantic segmentation network is configured to perform semantic segmentation on a second frame within a predetermined number of consecutive video frames to obtain at least one second semantic feature of the second frame, wherein the second frame follows the first frame; a third semantic segmentation network is configured to perform semantic segmentation on a third frame within the predetermined number of consecutive video frames to obtain at least one third semantic feature of the third frame, wherein the third frame is located after the second frame and also after the first frame via a first predetermined number of consecutive frames; And a combined network, configured to generate at least one fourth semantic feature of the second frame based on the at least one first semantic feature, the at least one second semantic feature, and the at least one third semantic feature, wherein the predetermined number of consecutive video frames includes a fixed number of consecutive video frames, wherein the first frame is at the beginning of the fixed number of consecutive video frames, wherein the third frame is at the end of the fixed number of consecutive video frames, wherein the second frame includes a single frame between the first frame and the third frame in the middle of the fixed number of consecutive video frames, and wherein the first frame and the third frame include key frames.

14. The system of claim 13, wherein the system is further configured to: propagate the at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; and propagate the at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature.

15. The system as described in claim 13, further comprising: A first optical flow network is configured to determine at least one first forward optical flow from a fourth frame to a second frame and at least one first backward optical flow from the second frame to the fourth frame, wherein the fourth frame is located between the first frame and the second frame, and the fourth frame is immediately adjacent to the second frame. And a second optical flow network configured to determine at least one second forward optical flow from the second frame to the fifth frame and at least one second backward optical flow from the fifth frame to the second frame, the fifth frame being between the second frame and the third frame, and the fifth frame being immediately adjacent to the second frame.

16. The system as described in claim 15, further comprising: A first masking network is configured to determine at least one first masking map from the fourth frame to the second frame and at least one second masking map from the fifth frame to the second frame, based at least in part on the at least one first forward optical flow, the at least one first backward optical flow, the at least one second forward optical flow, and the at least one second backward optical flow; and wherein the combined network is further configured to generate at least one third masking map of the second frame based on the at least one first masking map and the at least one second masking map.

17. The system of claim 16, wherein the system is further configured to: propagate the at least one first semantic feature toward the second frame in a frame-by-frame manner to obtain at least one first propagated semantic feature; and propagate the at least one third semantic feature toward the second frame in a frame-by-frame manner to obtain at least one second propagated semantic feature, and wherein the combined network is further configured to generate a semantic segmentation frame of the second frame based on the at least one first propagated semantic feature and the at least one first masking map, the at least one second propagated semantic feature and the at least one second masking map, and the at least one second semantic feature of the second frame and the at least one third masking map of the second frame by: multiplying the at least one first propagated semantic feature by the at least one first masking map by element-wise multiplication to obtain at least one first weighted semantic feature; multiplying the at least one second propagated semantic feature by the at least one second masking map by element-wise multiplication to obtain at least one second weighted semantic feature; At least one third weighted semantic feature is obtained by multiplying the at least one second semantic feature with at least one third masking mapping by element-wise multiplication; and the at least one first weighted semantic feature, the at least one second weighted semantic feature, and the at least one third weighted semantic feature are multiplied by element-wise summation.

Images