Faster r-cnn context mechanism optimization method based on multi-layer feature fusion

Abstract

The application relates to a Faster R-cnn context mechanism optimization method based on multi-layer feature fusion, and belongs to the technical field of target recognition. The method is based on the traditional use of HyperNet to extract a preselected frame feature vector, adds a multi-layer feature fusion module which is improved based on a mode different from the traditional mode into the model, extracts semantic information at different levels of the image, improves the traditional sampling mode into reverse pooling to avoid the chessboard problem, then fuses the obtained feature vector with a context attention mechanism to obtain a multi-information feature vector, and finally performs recognition and classification. The application can improve the accuracy of the model for recognizing a static object.

Description

Technical Field

[0001] This invention belongs to the field of target recognition technology and relates to a target detection method based on an optimized context mechanism (a Faster R-CNN context mechanism based on multi-layer feature fusion). Background Technology

[0002] In some high-altitude surveillance scenarios, passive surveillance cameras capture the environment in a static state for extended periods. The images captured by static cameras exhibit greater long-term consistency across each frame compared to those captured by dynamic cameras. Within the captured field of view, images from different months and days can also show significant similarities. For example, when analyzing images captured by rail transit cameras, the system attempts to use relevant information from a period closely preceding these images to help determine and assess the current situation. This is accomplished through attention mechanisms that incorporate past memory data.

[0003] Video object detection architectures are built upon single-frame models and address other issues in video data, such as motion blur, occlusion, and rare poses, by merging and fitting contextual cues from other frames (as in the traditional HyperNet approach). These methods employ a flow-like concept to aggregate features or use correlations to tightly link features at the current time step with those at adjacent time steps. Most of the aforementioned video detection methods are not well-suited for problems with sparse, irregular frame rates for object detection. For example, flow-based methods, 3D convolutions, and LSTMs typically employ denser, more regular frame rate sampling.

[0004] The traditional HyperNet approach involves a CNN extracting features from an input image to obtain corresponding feature maps. These maps are then fed into an RPN network to generate candidate bounding boxes, which are then classified. Classification involves determining whether the content within the candidate box is background or foreground content, discarding background candidates and retaining only foreground content. However, this traditional approach suffers from weak semantic information in shallow layers, failing to fully describe the representation of small targets. Upsampling is used to fuse feature maps with features from other layers, which not only leads to information loss and omission but also hinders the detection of small targets and impacts the algorithm's detection speed for different targets.

[0005] Deconvolution, an upsampling method in convolutional layers, is also known as transposed convolution. As a traditional convolution method, it can cause a checkerboard effect in the resulting image. Therefore, unpooling is used in upsampling to replace it, thus resolving the checkerboard effect.

[0006] Therefore, a new contextual mechanism optimization scheme is urgently needed to solve the problem of inaccurate target detection in existing static videos. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a Faster R-CNN contextual mechanism optimization method based on multi-layer feature fusion. This method utilizes the characteristic that the long-term position of a static camera does not change, employing a feature freezing method to create a long-term memory bank that stores the long-term habitual behaviors of the detected object. A short-term attention mechanism is added to the structure, incorporating both positive and false positive examples into the learning scope, correcting the recognition results, and improving the model's accuracy in recognizing stationary objects.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] An optimization method based on multi-layer feature fusion and a contextual attention mechanism for Faster R-CNN is proposed. Building upon the traditional method of extracting pre-selected bounding box feature vectors using HyperNet, this method incorporates a multi-layer feature fusion module, which differs from traditional approaches, to extract semantic information at different levels of the image. Furthermore, the traditional sampling method is improved by using unpooling to avoid the checkerboard pattern problem. The obtained feature vectors are then fused with a contextual attention mechanism to obtain multi-information feature vectors, which are ultimately used for recognition and classification.

[0010] The method specifically includes the following steps:

[0011] S1: Input the current keyframe and relevant context frames into the multi-layer feature fusion module to extract the preselection box feature vector;

[0012] S2: In the multi-layer feature fusion module, the deep convolutional layers use unpooling for upsampling;

[0013] S3: For the preselected bounding box feature vectors of the current keyframe and related context frames output by the multi-layer feature fusion module, input them into the attention module and the module for building short-term memory, respectively.

[0014] S4: The feature vector of the preselected box in the current keyframe is processed by the long short-term attention module to merge context-related features;

[0015] S5: Collect the processed feature vectors of the preselected boxes, and classify and optimize them.

[0016] Further, in step S1, the current frame is used as a keyframe, and together with the relevant frames before and after it, they are extracted and input into the multi-layer feature fusion module. The multi-layer feature fusion module is based on the improved HyperNet method and the VGG network. It extracts features at different depths of the convolutional layers in different ways to better combine the semantic information of each layer. Specifically, instead of selecting the traditional feature layers 1, 3, and 5, the module combines the features of layers 1, 2, and 5, and combines the deep features of layers 1, 4, and 5. Finally, the features of the two layers are concatenated to obtain more detailed semantic information. Then, the entire module forwards and generates clustered feature maps through the convolutional layers, and then compresses them into a unified space, i.e., super-features. Finally, a pre-selection box generation network is constructed to generate approximately 100 pre-selection boxes as output, returning the feature vectors of the pre-selection boxes extracted from each frame.

[0017] Furthermore, in step S1, the multi-layer feature fusion module receives the current input frame. Using VGGNet as its base network, the module extracts features from layers 1, 2, and 5 and concatenates them. For the features of layer 1, it first passes them through a convolutional layer to reduce the number of channels and thus reduce computation. Similarly, the features of layers 2 and 5 are passed through convolutional layers and then upsampled. Afterward, unpooling is used to make the pixels of the second and third layers identical to those of the first layer, facilitating fusion.

[0018] The same operation is applied to the deep features of layers 1, 4, and 5. For the features of layer 1, they are also passed through a convolutional layer to reduce the number of channels and thus reduce the amount of computation. Similarly, the features of layers 4 and 5 are passed through a convolutional layer and then upsampled. After that, unpooling is used to make the pixels of the features of layers 4 and 5 the same as those of layer 1, so as to facilitate fusion.

[0019] Furthermore, in step S2, the traditional deconvolution upsampling method used in the multi-layer feature fusion module is replaced with unpooling to solve the checkerboard problem caused by the former. Pooling discards unimportant information and retains the main information. Unpooling is the inverse operation of pooling; when restoring the original data value from the pooled data, a padding operation is used to fill in the missing data in the restored data, thereby maximizing the data integrity.

[0020] Furthermore, in steps S3 to S4, the feature vector of the current frame preselection box is passed through two attention modules based on different memory banks. These modules are indexed into the memory banks in different ways, so that the short-term memory bank incorporates the features of the context frame into this memory bank. These attention-based modules return a context-based feature vector, which is input into the next step for classification and optimization of the preselection box.

[0021] Furthermore, in step S3, the preselected bounding box feature vectors of the relevant context are input into the short-term memory module to construct a short-term memory library; the preselected bounding box feature vectors of the current keyframe are processed through the long and short-term memory modules respectively, and then summarized into feature vectors related to short-term and long-term features.

[0022] Further, in step S5, the feature vectors of the preselected bounding boxes of keyframes and related context frames processed by the attention module are obtained and fed into the classification module. Based on the traditional FC-Dropout-FC-Dropout hierarchical design, a 3×3×63 convolutional layer is added before the fully connected layer. The feature dimension is reduced by half by the convolutional layer before being input into the fully connected layer, optimizing the calculation of each layer. Secondly, the dropout ratio is reduced to half of its original value to reduce overfitting. In other words, the preselected bounding boxes are processed through a pooling layer followed by a convolutional layer; the rest is largely the same as the traditional model. The preselected bounding boxes are classified using pooling and convolutional layers to complete the object detection output.

[0023] The beneficial effects of this invention are as follows: First, this invention improves upon HyperNet in traditional models, specifically by enhancing its fusion within the contextual attention model to obtain more detailed semantic information. Second, this invention improves the upsampling method in traditional feature fusion. Combining these methods and applying them to high-altitude static camera recognition scenarios enhances the recognition of small targets and improves detection accuracy.

[0024] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0026] Figure 1 This is a model framework diagram of the improved HyperNet context method based on multi-layer fusion in this invention;

[0027] Figure 2 This is a flowchart of the improved multi-layer feature fusion module in the model of this invention;

[0028] Figure 3 This is a structural diagram of the attention module used in this invention;

[0029] Figure 4 This is a structural diagram of the preselection box classification and box optimization module;

[0030] Figure 5 This is a schematic diagram of the average pooling process;

[0031] Figure 6 This is a schematic diagram of the reverse average pooling process. Detailed Implementation

[0032] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] Please see Figures 1-6 This invention provides a target detection method based on the Faster R-CNN context mechanism with multi-layer feature fusion. A "memory" is constructed based on the context framework, and a detection model is modified to make predictions based on this memory.

[0034] like Figure 1 As shown, the improved HyperNet context method model structure based on multi-layer fusion is as follows:

[0035] First, the image extracts preselected bounding box features from the current frame and related frames through a multi-layer feature fusion module, and returns a set of class-independent preselected bounding box feature vectors for each frame. For example... Figure 2 As shown, the implementation steps of the multi-layer feature fusion module model are as follows:

[0036] The initial image is input into the feature extraction network, and different sampling strategies are applied to different layers. Layers 1, 2, and 5 are then fused after being processed using different feature extraction methods.

[0037] For the first layer, it is first passed through a 3x3 convolutional layer to reduce the number of channels in the feature layer, and then through a ReLU layer. The remaining two layers are first passed through a 1x1 convolutional layer to reduce the number of channels in the same way. After that, the features from the convolutional layers are fed into a ReLU layer for unpooling and upsampling. Finally, the three feature layers are fused together.

[0038] Analogous to the deep features of layers 1, 4, and 5, for the features of layer 1, we also pass them through a convolutional layer to reduce the number of channels and thus reduce the amount of computation. Similarly, the features of layers 4 and 5 are passed through a convolutional layer and then upsampled. After that, we use unpooling to make the pixels of these two feature layers the same as those of the first layer so as to facilitate fusion.

[0039] The entire module is then forwarded through convolutional layers to generate clustered, hierarchical feature maps, which are then compressed into a unified space, i.e., superfeatures. Next, a pre-selection bounding box network is built, producing approximately 100 pre-selection bounding boxes as output, completing the return of the vector set.

[0040] The pre-selected bounding box features returned after multi-layer feature fusion are indexed into memory by two attention-based modules (differently), allowing the model to merge features from context frames (seen by the same camera) to provide local and global temporal context. Specifically, the current frame's pre-selected bounding box feature vector is passed through two attention modules based on different memory banks, which are indexed into the memory in different ways. This allows the short-term memory to incorporate features from the context frames, providing both local and global temporal context.

[0041] like Figure 3 As shown, the structure of the attention module is as follows:

[0042] Keyframe features are tensors of the input features for the current frame, with a shape of [n×7×7×2048], where n is the number of candidate boxes returned by the multi-layer feature network. The keyframe features are spatially pooled along the feature width and height dimensions to produce a pooling layer of shape [n×2048]. The input to the relevant context feature matrix can be either short-term memory or long-term memory. We define k as the key function, q as the query function, v as the value function, and f as the final projection, which returns us to the correct output feature length and adds it back to the input features.

[0043] Using two different θ(θ) 长期 or θ 短期 The symbols ) represent long-term or short-term attention, respectively; A represents the input of keyframe features, and B represents the input of relevant context features; k, q, v, and f are all fully connected layers with an output dimension of 2048. Standard dot-multiplication attention is used to calculate the attention weight w:

[0044]

[0045] Where w is the attention weight of shape [n×m], and d is the feature depth (2048). Next, a context feature output F is constructed for each bounding box by projecting and weighting the context features. context By projecting and weighting the context features:

[0046] F context =f(w·v(B;θ);θ)

[0047] Among them, F contextThe shape is [n×2048]. Finally, F context It is added as a deviation for each feature channel to the original keyframe feature information.

[0048] These attention-based modules return a context-based feature vector, which is then fed into the next step for classification and optimization of the preselected bounding boxes, such as... Figure 4 As shown.

[0049] Contextual long-term memory: Given a predefined time range i t-k :i t+k Given a keyframe i on the object to be detected t We run a fixed, pre-trained detector on each frame. We construct a long-term memory (Mlong) from the feature vectors corresponding to the detected results. Due to hardware memory limitations, the content stored in the memory is critical. Therefore, the following two strategies are implemented to ensure the storage of the memory: We prune candidate boxes at the RPN to obtain instance-level feature tensors, and only save the spatial set representation of such tensors for each class, concatenating the date with the spatiotemporal encoding of the box position (generating the embedding vector for each box).

[0050] Limiting the number of candidate boxes—considering multiple strategies to ultimately decide which features and how many features to store in the memory. Adding a sparse frame rate to the above, and using these strategies, a context feature memory capable of holding up to 8500 elements can be constructed.

[0051] Short-Term Memory: Experiments show that using the same trained first-stage feature extractor as the keyframes, and adding a separate mechanism to incorporate short-term background features from nearby frames, is helpful. Unlike the previous-stage memory, we use a frozen feature extractor to build long-term memory over a longer time frame. The difference lies in that we do not refactor short-term features: for small window sizes, all candidate box window features can be retained in memory. A superimposed tensor of instance-level features is extracted from all frames within a small window surrounding the current frame and aggregated across the entire set in the spatial dimension. This produces a matrix of shape (number of candidate box frames per frame) * (feature depth), containing a single embedding vector (short-term memory) for each candidate box, which is then passed to the short-term attention module.

[0052] Pre-selection box classification and optimization: An improvement on the traditional detection network, adding a 3×3×63 convolutional layer before the fully connected layer, reducing the feature dimension by half. The pre-selected boxes are processed through an RIO pooling layer followed by an additional convolutional layer; the rest is similar to the traditional model. The final recognition result is obtained through scoring and regression operations. The pre-selection box module is as follows: Figure 4 As shown.

[0053] Unpooling: Unpooling is the inverse operation of pooling. The pooling process can only retain the main information while discarding some less important information; it cannot restore the initial data values ​​from the pooled data. If you try to restore the original data values ​​from the pooled data, the result will contain missing data. This can be addressed by using partial operations to complete the data and achieve a more comprehensive maximization operation.

[0054] Average pooling and anti-average pooling: such as Figure 5 and Figure 6 As shown, first restore the original size, then fill each value from the pooling result into its corresponding position in the original data region. The process of average pooling and inverse average pooling is as follows: Figure 5 .

[0055] Max pooling and unmax pooling: This requires recording the coordinates of the maximum activation value during pooling. In the subsequent unmax pooling process, only the coordinates of the maximum activation value are activated, while other values ​​are set to 0. This process is only an approximation because, during pooling, values ​​other than the maximum value are not zero.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A Faster R-CNN contextual mechanism optimization method based on multi-layer feature fusion, characterized in that, Building upon the traditional method of extracting preselected bounding box feature vectors using HyperNet, this method incorporates an improved multi-layer feature fusion module to extract semantic information from different levels of the image. The traditional sampling method is also improved by unpooling. Then, the obtained feature vectors are fused with a contextual attention mechanism to obtain multi-information feature vectors, which are ultimately used for recognition and classification. The method specifically includes the following steps: S1: Input the current keyframe and related context frames into the multi-layer feature fusion module to extract pre-selected bounding box feature vectors. Specifically, the current frame is used as a keyframe, and together with the relevant frames before and after it, they are extracted and input into the multi-layer feature fusion module. The multi-layer feature fusion module is based on the improved HyperNet method, which uses a VGG network to extract features at different depths of the convolutional layers. Specifically, it combines the features from layers 1, 2, and 5, and combines the deep features from layers 1, 4, and 5. Finally, it concatenates the features from the two layers. Then, the entire module forwards the data through convolutional layers to generate clustered and hierarchical feature maps, and then compresses them into a unified space, i.e., superfeatures. Finally, a pre-selected bounding box generation network is constructed to output the pre-selected bounding box feature vectors extracted from each frame. S2: In the multi-layer feature fusion module, the deep convolutional layers use unpooling for upsampling; S3: For the preselected bounding box feature vectors of the current keyframe and related context frames output by the multi-layer feature fusion module, input them into the attention module and the module for building short-term memory, respectively. S4: The feature vector of the preselected box in the current keyframe is processed by the long short-term attention module to merge context-related features; S5: Collect the processed feature vectors of the preselected boxes, and classify and optimize them; The feature vectors of the preselected bounding boxes in the keyframes and related context frames after processing by the attention module are obtained and fed into the classification module. Based on the traditional FC-Dropout-FC-Dropout hierarchical design, a convolutional layer is added before the fully connected layer; that is, a convolutional layer is added after the preselected bounding box passes through the pooling layer, and the rest is similar to the traditional model. The preselected bounding boxes are classified through pooling and convolutional layers to complete the target detection output.

2. The Faster R-CNN context mechanism optimization method based on multi-layer feature fusion according to claim 1, characterized in that, In step S2, the traditional deconvolution upsampling method used in the multi-layer feature fusion module is replaced with unpooling; when restoring the original data value from the pooled data, a padding operation is used to pad the missing data in the restored data.

3. The Faster R-CNN context mechanism optimization method based on multi-layer feature fusion according to claim 1, characterized in that, In steps S3-S4, the feature vector of the current frame preselection box is passed through two attention modules based on different memory banks. These modules index the memory banks in different ways, so that the short-term memory bank incorporates the features of the context frame into this memory bank. These modules return a context-based feature vector, which is input into the next step for classification and optimization of the preselection box.

4. The Faster R-CNN context mechanism optimization method based on multi-layer feature fusion according to claim 1 or 3, characterized in that, In step S3, the feature vectors of the preselected bounding boxes of the relevant context are input into the short-term memory module to build a short-term memory library; the feature vectors of the preselected bounding boxes of the current keyframe are processed by the long and short-term memory modules respectively, and then summarized into feature vectors related to short-term and long-term features.

Images