A video target detection method based on category-aware feature aggregation

Abstract

The application discloses a video target detection method based on category-aware feature aggregation, comprising: detecting a main stem ResNet-101 to perform feature extraction on a video frame to obtain high-level semantic information; using deformable convolution to perform sub-pixel level feature alignment, and then using a region proposal network RPN to generate a target candidate frame for each frame; using a candidate frame classification module to perform classification operation on the generated candidate frame, and subsequently performing feature aggregation only on candidate frames with the same class label; when performing category-aware feature aggregation, using a wide-range layer-by-layer progressive feature aggregation module to perform bidirectional and layer-by-layer progressive feature aggregation operation on the candidate frame level; using an inter-class relationship modeling module to model the spatial position relationship of different category targets on the same frame; and inputting the aggregated candidate frame features into a full connection layer to perform specific category discrimination and regression correction of the target frame position. The application fully utilizes the time sequence information and the spatial position information, thereby effectively improving the detection performance of the video target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video target detection method based on category-aware feature aggregation. BACKGROUND

[0002] Video target detection is a new research direction in the field of computer vision, and its main task is to correctly classify and locate the target in each frame of the video. Compared with the target detection task of static images, the appearance, shape and scale of the target will change with the movement of the target, so there are often detection difficulties such as motion blur, video defocus, partial occlusion and strange posture in the detection process. However, because video has more information in the time dimension than static images, how to use the time sequence information to make the target consistent in time order so that the target will not be lost in the frame with poor quality is the main difficulty of the video target detection task.

[0003] Most of the video target detection methods that have appeared in recent years are also exploring the possibility of improving detection accuracy brought by time sequence information, but there are still problems such as insufficient and inaccurate use of time sequence information, using unaligned feature extraction candidate boxes, not filtering time sequence information from adjacent frames, and ignoring the spatial context information of the target in the same frame. SUMMARY

[0004] In view of the above defects of the prior art, the technical problem to be solved by the present application is to provide a video target detection method based on category-aware feature aggregation. The method performs double feature aggregation operations in two aspects of feature level and instance level, aligns the features using inter-frame motion information, and improves the spatio-temporal robust consistent expression of frame features. In the instance level, bidirectional feature aggregation is performed layer by layer for the same category, which avoids the interference of noise information of irrelevant categories. Finally, the spatial position correlation of different category targets in the same frame is learned, more context information is obtained, and both the time sequence information and the spatial position information are fully utilized, thereby effectively improving the detection performance of the video target.

[0005] To achieve the above purpose, the present application provides a video target detection method based on category-aware feature aggregation, comprising:

[0006] A detection backbone ResNet-101 performs feature extraction on the video frame to obtain high-level semantic information, performs sub-pixel level feature alignment using deformable convolution, and then uses a region proposal network RPN to generate target candidate boxes for each frame;

[0007] A candidate box classification module is used to classify the generated candidate boxes, and subsequent feature aggregation is performed only on candidate boxes with the same class label.

[0008] In the process of category-aware feature aggregation, a wide-range layer-by-layer progressive feature aggregation module is used to perform a bidirectional and layer-by-layer progressive feature aggregation operation of the candidate frame level, and the intermediate results calculated in the front and back directions of the current frame are saved and used to update the support frames used in the calculation of the next frame;

[0009] An inter-class relationship modeling module is used to model the spatial position relationship of different class targets on the same frame;

[0010] The aggregated candidate frame features are input into a full connection layer for specific class discrimination and target frame position regression correction.

[0011] Further, before the classification operation of the generated candidate frame by the candidate frame classification module, the current frame is further subjected to sub-pixel level fine-grained feature alignment by a feature layer motion alignment module, specifically, in the deformable alignment, the deformable convolution is used to align F t+s , F t-s and F t with each other pixel point by point, where F t represents the feature map of the current frame, F t+s and F t-s represent the local frame feature maps, and the mathematical formula can be expressed as:

[0012]

[0013] where R={(-1,-1),(-1,0),…,(0,1),(1,1)} represents a regular grid of a 3x3 convolution kernel, the weight of the convolution kernel is represented by w, p k represents the k-th sampling offset in the convolution kernel, and the output represents the aligned feature, and Δp k is the motion offset predicted by a series of convolutions on F t+s and F t ;

[0014] Finally, the aligned features and of the current frame and each local frame are added element by element to obtain the final output F t ' of the module, i.e., the feature of the current frame after sub-pixel level fine-grained feature alignment.

[0015] Further, the candidate frame classification module classifies the candidate frame generated by the RPN into different classes or background, and then only the candidate frames with the same class label are subjected to instance-level feature aggregation.

[0016] Further, the wide-range layer-by-layer progressive feature aggregation module is used for bidirectional and layer-by-layer feature aggregation operation of the candidate box level, specifically:

[0017] For the aggregation of the candidate box, first, the relationship between the candidate boxes needs to be modeled, and given a set of candidate boxes B={B1, B2, …, B n}, wherein each candidate box B i contains semantic and position information, and it is assumed that the candidate box feature set enhanced by the relationship modeling is The aggregation enhancement process can be represented by the following formula:

[0018]

[0019] Wherein γ(·) is a general transformation function, S k,i represents the similarity of B k and B i at the candidate box level, and the calculation formula of the similarity is as follows:

[0020]

[0021] Wherein represents the semantic feature similarity between the boundary box B k and B i , wherein and ξ(·) are the network structure for extracting the semantic feature of the boundary box, is the dot product operation, and g k,i is the geometric similarity of the boundary box B k and B i , and the specific calculation formula is as follows:

[0022]

[0023] Wherein θ(·) is a general transformation function, and δ(·) is a position embedding operation, which embeds the original low-dimensional geometric similarity g k,i into a high-dimensional representation of a deep detection network, and the geometric information of the boundary box B i is taken as an example, which can be represented as B i =(x i ,y i ,w i ,h i ), wherein x i and y i are the center coordinates of the boundary box B i , and w i and h i represent the width and height of the boundary box, respectively.

[0024] The beneficial effects of the present application are:

[0025] (1) A dual feature aggregation method is proposed, which unifies feature-level aggregation and instance-level aggregation. Before generating the candidate box, i.e. at the feature level, fine-grained sub-pixel level feature alignment is performed using deformable convolution to make the feature map more accurate and rich in semantic information, enhancing the robust expression of spatial and temporal consistency of the feature, so as to generate a higher quality candidate region box, and the subsequent instance-level candidate box feature aggregation is more accurate and efficient.

[0026] (2) A class-aware bidirectional and layer-by-layer progressive efficient feature aggregation structure is proposed. Class-aware means that the candidate box is first classified, and then the more accurate candidate box feature aggregation is performed for each class, effectively avoiding the interference of irrelevant class noise information and redundant aggregation calculation. Bidirectional means that the selection of local frames considers both the forward part of the current frame and the backward part of the current frame, so that the temporal information of the previous and subsequent frames can be fully used. The use of layer-by-layer progressive aggregation is to consider that the adjacent frames in time sequence have more target consistency, so the use of layer-by-layer progressive method can make the aggregation more accurate and efficient. In addition, the intermediate features generated during forward and backward aggregation are used to update the support frame features used in the next frame aggregation, so that the long-term temporal information utilization is realized.

[0027] (3) When modeling the spatial relationship of different class candidate boxes in the same frame, a new geometric similarity calculation method is proposed, which makes the candidate box obtain more spatial context information.

[0028] The concept, specific structure and technical effects of the present application will be further described in conjunction with the drawings to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0030] As Figure 1As shown, this invention provides a video object detection method based on category-aware feature aggregation, which consists of a video object detection backbone and four modules. The detection backbone, ResNet-101, extracts features from video frames to obtain high-level semantic information, and then uses a Region Generation Network (RPN) to generate target candidate boxes for each frame. During this process, the processing of the current frame and local frames differs. After feature extraction, a feature layer motion alignment module is used to deformably align the features of the current frame and local frames, achieving fine-grained sub-pixel-level feature propagation. Subsequently, a candidate box classification module is used to classify the generated candidate boxes, and feature aggregation is only performed on candidate boxes with the same class label. During category-aware feature aggregation, a wide-range, progressively layered feature aggregation module is used to perform bidirectional and progressively layered feature aggregation operations at the candidate box level. The intermediate results calculated before and after the current frame are saved and used to update the supporting frames used for the next frame calculation. Furthermore, an inter-class relationship modeling module is used to model the spatial relationships between different categories of targets on the same frame. Finally, the aggregated candidate box features are input into a fully connected layer for specific category discrimination and target box location regression correction. Wherein:

[0031] like Figure 1 As shown in the FMA module, where F t F represents the feature map of the current frame. t+s F t-s This represents a local frame feature map. In deformable alignment, deformable convolution is used to separate F... t+s F t-s With F t Perform pixel-by-pixel alignment at position p. The mathematical formula can be expressed as:

[0032]

[0033] Where R = {(-1,-1),(-1,0),…,(0,1),(1,1)} represents a regular grid of 3×3 convolution kernels, and the weights of the convolution kernels are represented by w, p k Represents the k-th sampling offset in the convolution kernel, output Represents the aligned features, Δp k It is for F t+s With F t The motion offset is predicted by performing a series of convolutions.

[0034] Finally, the features are aligned between the current frame and each local frame. and By adding element by element, we obtain the final output F of the module. t ′, which is the feature after subpixel-level fine-grained feature alignment of the current frame.

[0035] In order to make the feature aggregation use the most relevant effective information as much as possible, a candidate box classifier is inserted before the feature aggregation module. The classifier classifies the candidate boxes generated by the RPN into different categories or background, and then only the candidate boxes with the same category label are aggregated at the instance level, effectively avoiding the interference of irrelevant class noise information, making the feature aggregation process more efficient.

[0036] In order to obtain more long-term timing information, the current frame will obtain relevant information from the global frame, the local frame, and the bidirectional support frame to perform feature aggregation. The global frame is randomly selected from the shuffled video frames; the local frame is the adjacent frames of the current frame; the support frame comes from the bidirectional memory module, wherein the forward part stores the intermediate aggregation features calculated by the previous frame, and the backward part is the part of the frame after a long interval from the current frame. In addition, considering that the frames adjacent in the video frame sequence have more target consistency, therefore, the progressive aggregation method can make the aggregation effect more sufficient and accurate, and the specific aggregation order is as shown in Figure 1 the wide-range progressive feature aggregation module.

[0037] For the aggregation of candidate boxes, first, the relationship between the candidate boxes needs to be modeled. Given a set of candidate boxes B = {B1, B2, …, B n}, wherein each candidate box B i contains semantic and position information, and assuming that the candidate box feature set enhanced by relationship modeling is , the aggregation enhancement process can be represented by the following formula:

[0038]

[0039] wherein γ(·) is a general transformation function, S k,i represents the similarity of B k and B i at the candidate box level, and the calculation formula of the similarity is as follows:

[0040]

[0041] wherein represents the semantic feature similarity between the boundary box B k and B i , wherein and ξ(·) are the network structure for extracting the semantic features of the boundary box, and is the dot product operation, and g k,i is the geometric similarity of the boundary box B k and B i , and the specific calculation formula is as follows:

[0042]

[0043] Where θ(·) is the general transformation function, and δ(·) is the position embedding operation, which transforms the original low-dimensional geometric similarity g... k,i Embedded into the high-dimensional representation of the deep detection network, with bounding box B i For example, its geometric information can be represented as B. i =(x i ,y i ,w i ,h i ), where x i ,y i For candidate box B i The center coordinates of the border, w i and h i These represent the width and height of the border, respectively. By utilizing geometric and appearance information, the information distortion problem caused by noise during the fusion process can be mitigated.

[0044] Since the candidate box feature aggregation processing method is performed, no feature aggregation is performed between candidate boxes of different categories on the same frame. However, since the target on the same frame has context information in spatial location, it is necessary to model the spatial relationship between candidate boxes of different categories on the same frame. Spatial topological relationship is established by embedding additional candidate box position and shape information. The specific calculation method is the same as formula (4).

[0045] The dataset used in the experiment was ImageNet ILSVRC2015-VID, which contains 30 basic categories. The training set has 3,862 video clips and a total of over 1.12 million frames.

[0046] After verification by experimental results, the innovative method proposed in this study has improved the detection effect. Table 1 shows the ablation experiment results, where FMA, WPFA, and CRM represent the three modules that can improve the detection effect. It can be seen that each module brings about an improvement in detection performance. Compared with the baseline method, this method improves the mAP by 1.8%.

[0047] Table 1 Ablation experimental results between different modules

[0048] Method FMA WPFA CRM mAP (%) (a) 81.8 (b) √ 82.6 (c) √ √ 83.0 (d) √ √ √ 83.6

[0049] Table 2 shows the comparison results between our method and the current State-of-the-art (SOTA) method. It can be seen that our method improves mAP by 0.7% compared to MEGA. Although the detection accuracy of our method is slightly inferior to CFA-Net, the detection speed is greatly reduced because CFA-Net uses a complex and computationally intensive feature aggregation method.

[0050] Table 2 Comparison of experimental results of other methods on the ImageNet VID dataset

[0051] Method Trunk Base detector mAP (%) FGFA ResNet-101 R-FCN 76.3 D&T ResNet-101 R-FCN 75.8 MANet [7] ]]> ResNet-101 R-FCN 78.1 SELSA [8] ]]> ResNet-101 Faster R-CNN 82.7 MEGA ResNet-101 Faster R-CNN 82.9 HVRNet ResNet-101 Faster R-CNN 83.2 CFA-Net [5] ]]> ResNet-101 Faster R-CNN 85.0 Our method ResNet-101 Faster R-CNN 83.6

[0052] From the results of Table 1 and Table 2, it can be seen that, after trade-off between precision and speed, the video target detection method based on class-aware feature aggregation proposed in the present application can achieve better results compared with existing methods.

[0053] The above describes preferred embodiments of the present application in detail. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative effort based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the prior art according to the concept of the present application shall be within the protection scope defined by the claims.

Claims

1. A video target detection method based on category-aware feature aggregation, characterized in that, include: The ResNet-101 backbone is used to extract features from video frames to obtain high-level semantic information. Deformable convolution is used for sub-pixel-level feature alignment. Then, the Region Generation Network (RPN) is used to generate target candidate boxes for each frame. The candidate box classification module is used to classify the generated candidate boxes, and feature aggregation is then performed only on candidate boxes with the same class label. When performing category-aware feature aggregation, a wide-range progressive feature aggregation module is used to perform bidirectional and progressive candidate box level feature aggregation operations. The intermediate results calculated before and after the current frame are saved and used to update the supporting frames used for the next frame calculation. Use the inter-class relationship modeling module to model the spatial relationships between different categories of targets on the same frame; The aggregated candidate box features are input into the fully connected layer for specific category discrimination and regression correction of the target box position; Before classifying the generated candidate boxes using the candidate box classification module, the method further includes: performing sub-pixel-level fine-grained feature alignment on the current frame using the feature layer motion alignment module. Specifically, in deformable alignment, deformable convolution is used to align the F-values... t+s F t-s F t Perform pixel-by-pixel alignment at position p, where F t F represents the feature map of the current frame. t+s F t-s The mathematical formula for representing a local frame feature map is as follows: Where R = {(-1,1),(-1,0),..,(1,1)} represents a regular grid of 3×3 convolution kernels, the weights of the convolution kernels are represented by w, and p k Represents the k-th sampling offset in the convolution kernel, output Represents the aligned features, Δp k It is F t+s With F t The motion offset is predicted by performing a series of convolutions; Finally, the features are aligned between the current frame and each local frame. and By adding element by element, we obtain the final output F of the module. t ′, which is the feature after subpixel-level fine-grained feature alignment of the current frame; The feature aggregation operation at the candidate box level using the wide-range, progressively layered feature aggregation module is specifically as follows: For candidate box aggregation, the first step is to model the relationships between candidate boxes. Given a set of candidate boxes B = {B1, B2, ..., B...} n }, where each candidate box B i Including semantic and locational information, assuming the enhanced candidate box feature set through relation modeling is... The polymerization enhancement process can be represented by the following formula: In the formula, γ(·) is a general transformation function; S k,i B k With B i The similarity of candidate boxes at different levels is calculated using the following formula: In the formula, Represents bounding box B k With B i Semantic feature similarity between them; ξ represents the network structure for extracting semantic features of bounding boxes; ⊙ represents the dot product operation; g k,i For bounding box B k With B i The geometric similarity is calculated using the following formula: In the formula, Here, δ is the general transformation function; δ is the position embedding operation, which transforms the original low-dimensional geometric similarity g... k,i Embedded into the high-dimensional representation of the deep detection network, with bounding box B i For example, its geometric information is represented as: B i =(x i ,y i ,w i ,h i ), x i ,y i For bounding box B i The center coordinates of the border, w i ,h i Representing bounding box B i Width and height.

2. The video target detection method based on category-aware feature aggregation according to claim 1, characterized in that, The candidate box classification module classifies the candidate boxes generated by RPN into different categories or backgrounds, and then performs instance-level feature aggregation only on candidate boxes with the same category label.

Images