Unsupervised video segmentation methods
By learning the semantic information relationship in video frames and strengthening the target edge information, the pyramid structure FCN and self-attention mechanism are adopted to solve the problems of similar-looking target segmentation errors in the existing technology and target loss in complex scenarios, achieving higher accuracy video segmentation effect.
Patent Information
- Application Number
- CN202110509393.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-05-11
AI Technical Summary
When processing similar and similar instance targets, existing video segmentation methods can easily divide multiple targets into one, and in complex scenarios, the targets are lost or wrongly split the background into targets, and fail to effectively utilize semantic information, resulting in edge processing. Rough, unable to distinguish similar appearance objects.
By learning the semantic information relationship in video frames, spreading semantic information and strengthening target edge information, the pyramid structure FCN extracts features, generates semantic vectors, combines the self-attention mechanism and pyramid fusion module to optimize the target segmentation effect, and extracts and Supervise enhances details and generates video frame instance segmentation masks.
It can better capture semantic fuzzy targets, improve the segmentation effect of targets similar appearance, enhance the target positioning and segmentation accuracy in complex scenarios, and expand its application in the field of image and video processing.
Smart Images

Figure CN113139502B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of video segmentation, can perform instance-level segmentation on any video foreground target without human intervention, involves relevant knowledge of video and image processing, and particularly relates to an unsupervised video segmentation method. Background Art
[0002] Unsupervised video segmentation has become increasingly important in recent years. Its main purpose is to segment the main moving objects in the video without human intervention. Unsupervised video segmentation is widely used in video reasoning, video understanding and other fields.
[0003] Unsupervised video segmentation can be divided into two main categories: detection and tracking based, and pixel-based matching and propagation based. The detection and tracking based method first detects the target frame of a single frame image, and then uses the feature similarity in the corresponding detection frame to find the continuity of different targets in the time domain. This method can handle targets of various scales well, but the wrong detection results have an irreversible impact on the subsequent segmentation. The 'Video instance segmentation tracking with a modified architecture' method proposed by Lin et al. at CVPR in 2020 assigns an instance vector to each detected target on the basis of detection, and completes the continuity of the video in the time domain by supervising and calculating the vector similarity. At the same time, this method adds an auxiliary iterative detection to better solve the problem of object occlusion. Luiten et al. proposed 'Unsupervised offline video objectsegmentation and tracking' at the WACV conference. This method mainly uses the detection results to generate long and short tracking segments. The tracking segments from short to long make this method better adapt to the tracking of long sequences of targets. Algorithms based on pixel matching and propagation can train the model end-to-end, avoiding the influence of intermediate results, but similar objects will have similar features, so this method will cause the objects to fail to connect in the time domain. The STemSeg model proposed by Athar at the ECCV conference can directly learn and match the pixel-by-pixel similarity of the complete video sequence, thereby completing the tracking of long sequences.
[0004] In addition, many inventions on video segmentation have been proposed in recent years. Patent CN202011124541.7 discloses a method for optimizing the accuracy and efficiency of video target detection, recognition or segmentation. This method uses the recognition results of adjacent frames to first predict the sub-region to be detected in the current frame, and then processes the sub-region to be detected containing the target object. It does not need to process irrelevant backgrounds, which greatly reduces the amount of processing required and improves the computing efficiency. In 2020, Li Yongjie et al. proposed a method and device for classifying foreground pixel depth information and foreground segmentation of video images in patent CN202010786958.3, which uses the depth map of the video frame to assist segmentation. In 2021, Chen Zuguo et al. disclosed patent CN202011227875.7, which uses the method of dual-channel convolution kernel and multi-frame feature fusion to overcome the shortcomings of traditional edge segmentation boundaries that are not closed and continuous, and achieve good image segmentation effects.
[0005] Although the current video segmentation algorithms have achieved good results, there are still several problems that need to be solved. First, most of the existing video segmentation methods can effectively solve the problems of targets with obvious motion and prominent appearance. However, when dealing with instance targets with similar and close appearances, it is easy to segment multiple targets into one. This error will affect the subsequent video frames. In addition, the range of video data is very wide and the scenes are complex. When the foreground moving object is similar to the background or there is severe occlusion, the target will be lost or the background will be mistakenly segmented into the target. Finally, the existing methods do not take into account the acquisition of semantic information and the distinction of appearance edges at the same time, which leads to the method with better target acquisition being very rough in edge processing. On the contrary, the method of matching details pixel by pixel lacks the semantic information of the object, resulting in the inability to distinguish objects with similar appearance. Summary of the invention
[0006] The technical problem to be solved by the present invention is: for a given arbitrary video sequence, each target in the video frame of the video sequence is segmented without any prior information. In addition, the present invention should also be able to better capture semantically ambiguous targets, such as targets with large scale changes in the video sequence, targets with similar colors and brightness to the background, etc. In addition, the video segmentation model of the present invention can also be widely used in the segmentation of various categories of targets.
[0007] In order to achieve the above object, the technical solution adopted by the present invention is:
[0008] The design principle of the present invention is: based on an observed conclusion: the semantic information of the video frame in the video sequence can better distinguish different targets from the background than the appearance information, and the semantic information of the target includes the relative position, color, scale, etc. of the target in the video frame. The present invention can optimize the segmentation effect of similar targets and better locate the target in complex scenes by learning the relationship between the semantic information of all targets in a single video frame and propagating the semantic information of each target between consecutive video frames. At the same time, the enhancement of the target edge information also guides the positioning and segmentation of the target. Therefore, the present invention enhances the details and optimizes the video segmentation results by gradually extracting the edge and supervising the edge.
[0009] An unsupervised video segmentation method comprises the following steps:
[0010] The first step is to generate the target semantic vector
[0011] 1.1) First, input a video frame I in a given video sequence, and use the classic pyramid structure FCN to extract object features to obtain a series of pyramid feature maps Among them, H i , W i and C represent the width, height and number of channels of the pyramid feature map of the i-th layer respectively. The bottom-up structure of FCN is widely used in vision-related deep learning fields, such as target detection, panoramic segmentation, instance segmentation, video segmentation, etc.
[0012] 1.2) Each pyramid feature map P output by FCN i The fused features are obtained by passing through a series of convolutional layers and upsampling to the same size as the shallowest layer and summing up:
[0013]
[0014] Among them, F unif To fuse features, ConvUp i It means that the i-th layer pyramid feature map has gone through multiple sets of convolutional layers and upsampling.
[0015] 1.3) Enhance the space and channel of the fusion features obtained in step 1.2) and mine the semantic information, so as to better guide the subsequent online learning of semantic vectors. unif , use formula (2) to calculate an attention map W:
[0016] W=Sig(Conv 1×1 (F unif )) (2)
[0017] Among them, Conv 1×1Represents a convolutional layer, Sig represents the Sigmoid function; the larger W is, the greater the probability that the position is the target position.
[0018] The fusion features are enhanced as follows under the guidance of the attention map:
[0019] F ch =F unif ⊙Sig(Ch avg (F unif ⊙W)) (3)
[0020] F att =Conv 3×3 (Concat(F ch ,W)) (4)
[0021] Formula (3) outputs F ch represents the channel enhancement feature, where Ch avg represents the channel average operation. Formula (4) outputs F att Represents the spatial enhancement feature, where Conv 3×3 represents the convolutional layer, and Concat represents channel concatenation.
[0022] 1.4) Finally, spatial and channel enhancement features F att After four identical combined structures (convolution layer + group normalization + nonlinear activation function), the semantic vector feature F is output IE . Semantic vector feature F IE Each pixel point (a semantic vector) in the video frame is a rough semantic expression of a target or background in the video frame. Compared with the propagation of appearance pixels, the present invention makes more use of the relationship of the semantic global features (semantic vector features) of the target rather than the similarity of local appearance.
[0023] Step 2: Online Learning of Semantic Vectors
[0024] The initial semantic vector feature F IE There is only information about each video frame itself. When the target in the video frame is severely occluded or similar to the background, the target positioning will be blurred. At this time, the target will be lost or the background object will be mistakenly detected as the target. Since the semantic vector representations of the same target in different frames in each video sequence are similar and different from other targets and backgrounds, the present invention uses the propagation of each target semantic vector in the time domain to guide the learning of the semantic vector in the target positioning blurred video frame, so as to better locate the target. At the same time, in order to prevent the accumulation of errors in past information, the self-enhanced semantic vector features are also integrated to weigh the correctness of the propagation in the time domain. In particular, for the case where there is no time domain propagation in the first frame of each video sequence, go directly to the third step to generate a semantic mask. When processing subsequent frames of the video sequence, the previously formed semantic mask will be used. Specifically as follows:
[0025] 2.1) For each video frame t, firstly integrate the position information into the semantic vector feature F IE ; Here, the normalized coordinates coord∈[-1,1] are used to represent the position information, which are respectively related to the semantic features of the video frame at the current time T and past video frame semantic vector features Perform channel splicing, so that the position information can be better distinguished between adjacent instances during the online learning of semantic vectors. For past video frames, additional splicing is performed on the previously generated semantic mask. For past video frames, self-attention semantic features are generated in the following way
[0026]
[0027]
[0028] Among them, A t and M t Represent the semantic alignment features and self-attention map of the past video frames respectively; O t A semantic mask representing the prediction of past video frames; Represents the semantic vector features after filtering of past video frames.
[0029] The self-attention mechanism described by formulas (5) and (6) is used to filter out misleading information of past video frames.
[0030] 2.2) In order to make full use of historical information, after processing each frame, the semantic vector features of all video frames at time 0≤t<T are filtered Will be dynamically updated to a fixed-size memory pool Mem for the current time T T :
[0031]
[0032] Wherein, T represents the current time.
[0033] Formula (7) shows that each frame of the past time 0≤t<T video frame has the same effect on the current time T video frame in the propagation process instead of relying more on the adjacent frames.
[0034] For the current video frame, there is:
[0035]
[0036] Among them, A T and M T They represent the semantic alignment features and self-attention map of the current video frame respectively.
[0037] Considering that the motion of each target in the video sequence changes slowly, when the memory pool Mem is given T and the alignment feature A of the current video frame T Finally, the present invention fuses and matches the semantic vector of each position with a series of adjacent semantic vectors of different distances through the following pyramid fusion module (ASPP), and outputs the matched time domain propagation semantic features: Compared with pixel-by-pixel matching, semantic vector matching greatly saves computational effort.
[0038]
[0039] In addition, the present invention adds a self-attention mechanism to the semantic vector feature of the current video frame to obtain a self-enhanced semantic vector feature:
[0040]
[0041] 2.3) In order to prevent error accumulation during time domain propagation, the present invention fuses the semantic features of time domain propagation through concatenation and convolution layers and self-enhanced semantic vector features The semantic information after time domain propagation is integrated into the semantic information of the current video frame to highlight the target itself:
[0042]
[0043] Output fusion semantic vector features Increasing the gap between the target corresponding semantic vector and the background semantic vector can better highlight the semantic information of each target in the current video frame.
[0044] The third step is to enhance the independence of semantic vectors.
[0045] In the process of generating and learning the target semantic vector features in the first and second steps, the convolution deformation operation may cause some semantic vectors to contain information of multiple adjacent targets (especially objects that are heavily occluded). The semantic vector covering multiple target information will weaken the semantic vector containing only one of the targets, resulting in target loss or positioning errors. Therefore, the present invention designs a residual module to correct and highlight the semantic vector that can represent a single target, thereby improving the accuracy of target positioning and segmentation. The implementation process of the residual module is as follows:
[0046]
[0047]
[0048]
[0049] The pyramid fusion module ASPP in formula (12) can be established The relationship between each semantic vector and the surrounding semantic vectors at different distances, and the output semantic vector features Contains the redundant information of each semantic vector, and passes through a convolutional layer Conv in formula (13) 1×1 The weight map α obtained by using the Sigmoid function Sig represents the redundancy of the semantic vector corresponding to each position point. The higher α is, the more redundant information the semantic vector of the corresponding position contains. Therefore, formula (14) can weaken the semantic vector with more redundancy and highlight the semantic vector of a single target to obtain the filtered semantic vector feature. at last, The target position in the video frame is predicted through two convolutional layers with 3×3 convolution kernels and a Sigmoid function to obtain the semantic mask at the current time T, as shown in formula (15).
[0050]
[0051] Output prediction O T is a weight map, where the higher the weight, the higher the probability that the position represents a single target.
[0052] The fourth step is to generate detail extraction and video frame instance segmentation masks guided by edge refinement
[0053] In the third step, the semantic mask generated by the semantic vector feature represents the probability of a single target at each position, and the convolution output of the base feature and the dynamic convolution kernel is used to generate the target segmentation mask corresponding to each position. The dynamic convolution kernel K is composed of the fusion feature F unif After four identical combined structures (convolution layer + group normalization + nonlinear activation function), the detailed information of the base features directly affects the accuracy of the output target segmentation mask. Therefore, the base features are generated by unif Generated after detail enhancement.
[0054] Edges are an important clue for video segmentation and all segmentation-related tasks. Fully utilizing edge information can enhance the details of the underlying features and thus improve the accuracy of target segmentation in some difficult scenarios, such as the segmentation of the joints when multiple targets are connected. Therefore, the present invention firstly integrates the fusion feature F unif Perform dimensionality reduction to obtain dimensionality reduction feature F rdu , and then gradually extract edge information through formulas (16) and (17):
[0055] B str =Conv 3×3 (Deconv 3×3 (F rdu )) (16)
[0056] B sub =Conv 3×3 (Deconv 3×3 (B str )) (17)
[0057] Deconv in formula (16) 3×3 Denotes deconvolution, which has the function of restoring the image, so it can restore the encoded dimensionality reduction feature F to a certain extent rdu The details are then enhanced through a convolutional layer to obtain the rough detail feature B. str Formula (17) uses the same structure to further recover the detail information in the feature and obtains the detail feature B sub The above two detail features and dimensionality reduction features are fused through formula (18) to obtain edge features:
[0058] F bdry =Conv 3×3 (B str +Conv 1×1 (Concat(B sub ,F rdu ))) (18)
[0059] F bdry Under the supervision of the edge mask truth, learn the fusion feature F unif edge information.
[0060] Then the edge feature F bdry And the fusion feature F unif Combine to enhance fusion feature details and generate base features
[0061]
[0062] Among them, ε represents a set of 3×3 convolution, ReLU function and 1×1 convolution. After convolution with the dynamic convolution kernel K, the corresponding O(O T The general representation of ) segmentation prediction for each position:
[0063]
[0064] Finally, the segmentation prediction m and semantic mask O are obtained, and the final segmentation result is output through the existing technology Matrix NMS.
[0065] The beneficial effects of the present invention are:
[0066] The unsupervised video segmentation method provided by the present invention can locate the target by mining the semantic information of the target and the background in all video frames in the same video sequence in the spatiotemporal domain at the same time, can better handle the distinction of targets with similar appearance and the detection of targets in complex backgrounds, and finally improve the accuracy of target segmentation by strengthening the edges of the base features. At the same time, the present invention can also be well extended to other fields of image and video processing, such as image completion, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 It is the system block diagram.
[0068] Figure 2 (a) is the corresponding position relationship between the object (background) and the semantic vector in the video frame, (b) is the vector corresponding to the foreground instance in the semantic feature, and (c) is the corresponding position relationship between the instance mask and the semantic vector.
[0069] Figure 3 (a) is the complete result, (b) is the result without temporal information propagation, (c) is the result without single-frame independence enhancement, and (d) is the result of removing both temporal and spatial enhancements. DETAILED DESCRIPTION
[0070] The present invention is further described below in conjunction with specific embodiments.
[0071] An unsupervised video segmentation method comprises the following steps:
[0072] Step 1: The present invention first inputs a lawn activity scene of two dogs and a person ( Figure 2 (a)) video frame I is scaled to 480×864×3, and the object features are extracted using the classic pyramid structure FCN to obtain a series of pyramid feature maps in and C=256 represent the width, height and number of channels of the i-th pyramid feature map, respectively.
[0073] Step 2: Each pyramid feature map P output by FCN i Through formula (1), a series of convolutional layers are respectively passed and upsampled to the same size as the shallowest layer and then summed to obtain the fusion feature
[0074] Step 3: The fusion feature F obtained in step 2 unif , calculate an attention map through formula (2) The larger W is, the greater the probability that the position is the target position. Under the guidance of the attention map, the fused feature is first enhanced by the channel of formula (3). The channel average operation in the formula will generate a vector with the same dimension as the number of channels C = 256. After passing the Sigmoid function, the vector is multiplied by the fused feature to enhance the channel weight. Then, the spatial enhanced feature F is obtained by formula (4). att Step 4: After passing through four identical combined structures (convolution layer + group normalization + nonlinear activation function), the semantic vector features are output Each pixel (256-dimensional vector) represents a semantic vector. Figure 2 (a) The person and dog in the middle and the dog on the far right correspond to Figure 2 The three pixels in the middle and on the right in (b) are different from the other colors (semantic vectors).
[0075] Step 5: For the case where there is no temporal propagation in the first frame of each video sequence, go directly to step 9 to generate a semantic mask. When processing subsequent frames of the video sequence, the previously generated semantic mask will be used.
[0076] Step 6: For each video frame t, merge the position information into the semantic vector feature F IE ; Here we use normalized coordinates To represent the location information of the semantic vector feature, coord uses two channels to represent the horizontal coordinate coord x and the ordinate coord y , for the position (i,j) in the feature, we have
[0077]
[0078]
[0079] coord respectively corresponds to the semantic features of the video frame at the current time T and past video frame semantic vector features For the past video frames, the previously generated semantic mask is additionally spliced. For the past video frames, the self-attention semantic features are generated using formulas (5) and (6) to filter out misleading information from past video frames.
[0080] Step 7: After processing each frame, the semantic vector features of all video frames at past time 0≤t<T are filtered It will be dynamically updated into a fixed-size memory pool for the current time T through formula (7)
[0081] For the current video frame, the alignment feature is generated by formula (8): and self-attention map When a memory pool Mem is given T and the alignment feature A of the current video frame T After that, the present invention uses the pyramid fusion module (ASPP) of formula (9) to fuse and match the information of each semantic vector with a series of adjacent semantic vectors of different distances, and outputs the matched time domain propagation semantic features. ASPP is composed of dilated convolutions with different dilated intervals, and the dilated interval is set to {1, 2, 4}. In addition, the self-attention mechanism can be added to the semantic vector feature of the current video frame through formula (10) to obtain the self-enhanced semantic vector feature
[0082] Step 8: Fusion of temporal propagation semantic features through concatenation and convolutional layers and self-enhanced semantic vector features The semantic information after time domain propagation is fused into the semantic information of the current video frame to highlight the target through formula (11) and the fused semantic vector feature is output Figure 3 The results in columns (b) and (d) lack the semantic vector learning of this step. Although the video frames in the first row of the two columns segment each target well, as time goes by, errors such as multiple targets being segmented into one target mostly occur, such as the two people on the right side of (b), the three people in the third row of (d), and the two people on the right side of the fourth row are all segmented into one whole.
[0083] Step 9: Fusion of semantic vector features The pyramid fusion module ASPP uses formula (12) to establish the relationship between each semantic vector and the semantic vectors at different distances around it. The hole interval of ASPP is {1, 2, 3}, and the output semantic vector feature Contains the redundancy information of each semantic vector, and then After a convolutional layer Conv in formula (13) 1×1 The weight map α obtained by using the Sigmoid function Sig represents the redundancy of the semantic vector corresponding to each position point. The higher α is, the more redundant information the semantic vector of the corresponding position contains. Therefore, formula (14) can weaken the semantic vector with more redundancy and highlight the semantic vector of a single target to obtain the filtered semantic vector feature. at last, Formula (15) is used to predict the target position in the video frame using two convolution layers with 3×3 convolution kernels and a Sigmoid function, and the output prediction O T is a weight map, where the higher the weight, the higher the probability that the position represents a single target. Figure 3 There is no filtering process in (c) and (d) in this step, so the training process will be O(O) for the semantic mask. T The supervision of ) uses focal loss. Figure 2 Indicates the correspondence between the target in the video frame and the semantic vector feature. The video frame (a) is divided into 5×5 grids. The grids with a larger target ratio (the man and the dog in the middle, and the dog's tail on the right) correspond to the semantic vectors of the target in (b) (the two grids in the middle and the grid on the right). After learning the semantic vector, the target grid prediction and the corresponding target segmentation mask are finally obtained in (c).
[0084] Step 10: For the position O of a single target generated in step 9, the target segmentation mask corresponding to each position is generated using the convolution output of the base feature and the dynamic convolution kernel. The dynamic convolution kernel By fusion feature F unif After four identical combined structures (convolution layer + group normalization + nonlinear activation function), the detailed information of the base features directly affects the accuracy of the output target segmentation mask. Therefore, the base features are generated by unif Generated after detail enhancement.
[0085] The present invention firstly combines the fusion feature F unif Perform convolution dimensionality reduction to obtain the dimensionality reduction feature F rdu , and then gradually extract edge information through formula (16) (17). Deconv in formula (16) 3×3 Denotes deconvolution, which has the function of restoring the image, so it can restore the encoded dimensionality reduction feature F to a certain extent rdu The details are then enhanced through a convolutional layer to obtain the rough detail feature B. str Formula (17) uses the same structure to further recover the detail information in the feature and obtains the detail feature B sub The two detail features and the dimension reduction feature are fused through formula (18) to obtain the edge feature F bdry . F bdry Under the supervision of the edge mask truth, learn the fusion feature F unif Then the edge feature F bdry And the fusion feature F unif Combined with formula (19), the fusion feature details are enhanced and the base features are generated. Where ε represents a set of 3×3 convolutions, ReLU functions and 1×1 convolutions, and the output The number of channels is the same as the dynamic convolution kernel. After convolution with the dynamic convolution kernel K through formula (20), the segmentation prediction corresponding to each position of O is obtained. Finally, the segmentation prediction m and semantic mask O are output through Matrix NMS to output the final segmentation result.
[0086] Step 11: During the training process, for the edge feature F bdry The supervision is performed by generating edge masks
[0087] bdry=F bdry *K (21)
[0088] To complete. The supervision method for segmentation mask and edge mask prediction is shown in formula (22)(23).
[0089]
[0090]
[0091] Where L represents the loss value N pos represents the number of positive samples, k represents the corresponding position of each semantic vector, 1 is a judgment equation, when p k >0, 1 otherwise 0. m k represents the predicted segmentation (edge) mask, t k Represents the true value. i represents each pixel in the segmentation mask.
[0092] Step 12: During the test, the semantic mask O and the segmentation mask m are updated through Matrix NMS. The algorithm of Matrix NMS is shown in formulas (24)-(26):
[0093]
[0094]
[0095] f(D(m i ,m j ))=1-D(m i ,m j ) (26)
[0096] Where m i and m j represent different prediction masks respectively, D is the same as formula (23), o i and j Represents two different positions in the semantic mask O, decay j It indicates the suppression rate of the predicted j-th segmentation mask. The higher the suppression rate, the lower the possibility that the segmentation mask will be output as the result. Finally, the segmentation mask at the position where the semantic mask probability is less than 0.1 is removed to obtain the final video frame segmentation result (segmentation mask).
[0097] The above-described embodiments merely express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. An unsupervised video segmentation method, characterized in that: The following steps are involved: The first step is to generate the target semantic vector 1.1) First, input a video frame I in a given video sequence, extract the object features and obtain a series of pyramid feature maps Among them, H i , W i and C represent the width, height and number of channels of the i-th pyramid feature map respectively; 1.2) Each pyramid feature map P i The fused features are obtained by passing through a series of convolutional layers and upsampling to the same size as the shallowest layer and summing up: Among them, F unif To fuse features, ConvUp i Indicates that the i-th layer pyramid feature map has undergone multiple sets of convolutional layers and upsampling; 1.3) Enhance the space and channel of the fusion features obtained in step 1.2) and mine the semantic information to provide guidance for the subsequent online learning of semantic vectors; use the fusion feature F unif , use formula (2) to calculate an attention map W: W=Sig(Conv 1×1 (F unif ))(2) Among them, Conv 1×1 represents a convolutional layer, Sig represents the Sigmoid function; The fusion features are enhanced as follows under the guidance of the attention map: F ch =F unif ⊙Sig(Ch avg (F unif ⊙W)) (3) F att =Conv 3×3 (Concat(F ch ,W))(4) Formula (3) outputs F ch represents the channel enhancement feature, where Ch avg represents the channel average operation; the output of formula (4) is F att Represents the spatial enhancement feature, where Conv 3×3 represents the convolutional layer, and Concat represents channel concatenation; 1.4) Finally, spatial and channel enhancement features F att After four identical combination structures, the semantic vector feature F is output IE ; Semantic vector feature F IE Each pixel in is a rough semantic expression of an object or background in the video frame; Step 2: Online Learning of Semantic Vectors The propagation of each target semantic vector in the time domain is used to guide the learning of the semantic vector in the blurred video frame of the target positioning, so as to better locate the target; at the same time, in order to prevent the accumulation of errors in past information, the self-enhanced semantic vector features are also integrated to weigh the correctness of the propagation in the time domain; for the case where there is no time domain propagation in the first frame of each video sequence, it directly goes to the third step to generate a semantic mask; when processing subsequent frames of the video sequence, the previously generated semantic mask is used; the details are as follows: 2.1) For each video frame t, firstly integrate the position information into the semantic vector feature F IE ; The normalized coordinates coord∈[-1,1] are used to represent the position information, which are respectively related to the semantic features of the video frame at the current time T and past video frame semantic vector features Perform channel splicing. During the online learning of semantic vectors, the position information is used to better distinguish adjacent instances. For past video frames, the previously generated semantic masks are spliced. For the past video frames, self-attention semantic features are generated in the following way Among them, A t and M t Represent the semantic alignment features and self-attention map of the past video frames respectively; O t A semantic mask representing the prediction of past video frames; Represents the semantic vector features after filtering of past video frames; The self-attention mechanism described by formulas (5) and (6) is used to filter out misleading information of past video frames; 2.2) In order to make full use of historical information, after processing each frame, the semantic vector features of all video frames at the past time 0≤t<T are filtered Will be dynamically updated to a fixed-size memory pool Mem for the current time T T : Wherein, T represents the current time; For the current video frame, there is: Among them, A T and M T Represent the semantic alignment features and self-attention map of the current video frame respectively; When a given memory pool Mem T and the alignment feature A of the current video frame T Finally, the following pyramid fusion module ASPP is used to fuse and match the semantic vector of each position with a series of adjacent semantic vectors at different distances, and output the matched time domain propagation semantic features: In addition, a self-attention mechanism is added to the semantic vector feature of the current video frame to obtain a self-enhanced semantic vector feature: 2.3) To prevent error accumulation during time domain propagation, the semantic features of time domain propagation are fused through concatenation and convolution layers. and self-enhanced semantic vector features The semantic information after time domain propagation is integrated into the semantic information of the current video frame to highlight the target itself: The third step is to enhance the independence of semantic vectors. A residual module is designed to correct and highlight the semantic vector that can represent a single target, thereby improving the accuracy of target positioning and segmentation. The implementation process of the residual module is as follows: The pyramid fusion module ASPP in formula (12) can be established The relationship between each semantic vector and the surrounding semantic vectors at different distances, and the output semantic vector features Contains the redundancy information of each semantic vector. The weight map α obtained by formula (13) represents the redundancy of the semantic vector corresponding to each position point. Formula (14) can weaken the redundant semantic vectors and highlight the semantic vector of a single target to obtain the filtered semantic vector feature. at last, Predict the target position in the video frame and obtain the semantic mask at the current time T, as shown in formula (15): Output prediction O T is a weight map, where the higher the weight, the higher the probability that the position represents a single target; The fourth step is to generate detail extraction and video frame instance segmentation masks guided by edge refinement In the third step, the semantic mask generated by the semantic vector feature represents the probability of a single target at each position, and the convolution output of the base feature and the dynamic convolution kernel is used to generate the target segmentation mask corresponding to each position. The dynamic convolution kernel K is composed of the fusion feature F unif After four identical combined structures are generated, the F unif After enhancement, generate; First, the fusion feature F unif Perform dimensionality reduction to obtain dimensionality reduction feature F rdu , and then gradually extract edge information through formulas (16) and (17): B str =Conv 3×3 (Deconv 3×3 (F rdu ))(16) B sub =Conv 3×3 (Deconv 3×3 (B str ))(17) Public, Deconv 3×3 represents deconvolution; B str Indicates coarse and detailed features, feature B sub Represents detailed features; The above two detail features and dimensionality reduction features are fused through formula (18) to obtain the edge feature F bdry , F bdry Under the supervision of the edge mask truth, learn the fusion feature F unif Edge information; F bdry =Conv 3×3 (B str +Conv 1×1 (Concat(B sub ,F rdu )))(18) Then the edge feature F bdry And the fusion feature F unif Combine to enhance fusion feature details and generate base features Among them, ε represents a set of 3×3 convolution, ReLU function and 1×1 convolution; After convolution with the dynamic convolution kernel K, the corresponding O T Segmentation prediction for each position: The final segmentation prediction m and semantic mask O T , output the final segmentation result.
Citation Information
Patent Citations
Video image foreground pixel depth information classification and foreground segmentation method and device
CN111882576A
Dynamic video image segmentation method based on dual-channel convolution kernel and multi-frame feature fusion
CN112308082A
Method for optimizing precision and efficiency of video target detection, recognition or segmentation
CN112417963A
Semi-supervised video target segmentation method based on modulation network and feature attention pyramid
CN110163875A
Image segmentation method and device, image segmentation model training method and device and storage medium
CN111429460A
