A panoramic image target detection algorithm based on feature enhancement

Abstract

With the wide use of panoramic vision in virtual reality, unmanned driving, video surveillance and other fields, the demand for target detection in panoramic images is growing. However, panoramic images have large scale changes and object distortion characteristics, and existing detection methods trained by pinhole images cannot be effectively applied to panoramic images. This patent proposes a feature enhancement target detection method based on panoramic images. We construct a multi-scale feature guide module to capture multi-scale object semantic information through different receptive fields, and combine attention guide to build more effective global context modeling to improve the network's ability to perceive and represent different scale features of the target. At the same time, a feature space correction module is also proposed to correct the sampling position of the feature according to the context information to better adapt to the changes of distorted and twisted objects. A large number of experiments prove the effectiveness of the model in panoramic image target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision target detection technology, particularly to panoramic image target detection, and more specifically to a panoramic image target detection algorithm based on feature enhancement. Background Technology

[0002] With the development of society and technology, images are presented to people in increasingly rich forms to meet the needs of different application scenarios. In this context, panoramic images, as a special form of image, possess richer geometric information, higher resolution, and a wider field of view than traditional pinhole images. They are widely used in virtual reality, autonomous driving, video surveillance, and other fields. Therefore, accurate detection and recognition of targets in panoramic images has become an important requirement. When performing target detection on panoramic images, the panoramic image is generally projected onto a 2D plane and then unfolded. This results in distortion of objects and significant differences in target scale in the projected panoramic image. Traditional deep learning-based target detection algorithms are generally designed for ordinary pinhole images. These algorithms struggle to extract complete information about targets in panoramic images, leading to bounding boxes that cannot adequately enclose deformed and extended targets, resulting in low detection accuracy. Currently, among target detection algorithms for panoramic images, Wang et al. introduced multi-kernel layers based on the RCNN model to improve the accuracy of detecting distorted objects. Jia et al. used deformable convolutions in the backbone of the YOLOv4 network to prevent geometric distortion of targets and added lateral skip connections in the feature fusion part to enhance feature fusion, achieving effective detection of panoramic images. These methods all process distorted targets in the feature extraction part, which introduces a large number of parameters, increasing model complexity. Furthermore, they do not consider that at different scales, the sampling operation will lose details of distorted target boundaries during feature fusion, leading to inaccurate target detection. He et al. addressed the problem of significant changes in object size as the field of view increases in panoramic images by proposing a method of segmenting the panoramic image, detecting targets in different slices through cascading, and finally merging the different slices using NMS. However, this method has insufficient modeling of slice targets and lacks processing for distorted targets. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a multi-scale feature guidance module and a feature space correction module, constructing a feature enhancement network more suitable for panoramic image object detection. The effectiveness of our method is validated on a panoramic image dataset. Compared with other mainstream object detection algorithms, the proposed model not only achieves the most accurate results but also boasts lower floating-point operations per second (FPS). Our method can not only promote the development of future panoramic image target detection technology, but also be used as a benchmark or comparative test to accelerate the application of panoramic vision-based artificial intelligence in target detection. In object detection, global context information, encompassing the background information of the entire scene, helps to more accurately locate and identify objects at different scales. Due to the imaging and projection methods of panoramic image features, panoramic images contain objects with very large scale variations, and previous detection algorithms have relatively limited capabilities in handling multi-scale changes in targets. Therefore, we propose a multi-scale feature guidance module. This module adaptively fuses multi-scale features in deep semantic features through multiple receptive fields, and combines attention guidance to reduce semantic conflicts during feature fusion, constructing a more effective global context model and improving the network's perception and representation capabilities for target features at different scales. When fusing features at different scales to extract information about distorted targets, features obtained through continuous sampling operations lose details of the distorted target boundaries, causing the model to fail to effectively extract complete information about distorted targets during multi-scale fusion, ultimately resulting in missed or partial detection of distorted objects in panoramic images. To address this, we propose a feature space correction module, which uses context information provided by local features to correct the position of sampled features, effectively capturing changes in the shape of distorted targets within multi-scale features and improving the network's shortcomings in adaptive feature fusion. Attached Figure Description

[0004] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are the modular structure of the present invention. Figure 1 It is the overall structure of the feature-enhanced panoramic image target detection model, with the multi-scale feature guidance module and the feature space correction module as the main components of the model. Figure 2 It is the overall architecture of the multi-scale feature guidance module. It obtains multi-scale features through dilated convolutions with different dilation rates, and constructs more effective global context modeling through channel and spatial attention guidance. Figure 3 It is the overall structure of the feature space correction module. It uses spatial enhancement to correct the position of the sampled features based on the context information provided by the local features through deformable convolution, thereby improving the shortcomings of the network in the feature fusion part. Detailed Implementation

[0005] The specific embodiments of the present invention will now be further described with reference to the accompanying drawings: Figure 1This demonstrates our invented feature-enhanced panoramic image target detection network model. After feature extraction, the model introduces a multi-scale feature guidance module, which improves the network's perception and representation capabilities for multi-scale targets by constructing effective global context modeling. In the feature fusion part, a feature space correction module is introduced, which uses context information to assist in the position correction of sampled features, effectively capturing the shape changes of distorted targets, thereby improving the shortcomings of the network's feature fusion part. Figure 2 As shown, the multi-scale feature guidance module takes the features extracted from the backbone network as input, and then, after simple processing, obtains features at different scales through dilated convolutions with different dilation rates and residuals. Finally, it concatenates and fuses the multi-scale features with the initial features as input for the second half of the process. Here, Convs represents the combined use of 1x1 and 3x3 convolutions. This represents a dilated convolution with a 3x3 kernel and a dilation rate of i (i=1,3,5). Then, max pooling and average pooling are used to generate weights for each channel, while the Softmax function is used to generate spatial dimension adaptive weights. Afterwards, the concatenated features are multiplied by the channel weights and spatial weights respectively, and then summed together to guide the features to learn more critical aspects, resulting in more refined global features. Where ⨂ represents the dot product operation, and AP and MP represent the average pooling and max pooling operations, respectively. Figure 3 As shown, the module first applies SGE attention to the large-scale features of the input. and small-scale features Perform spatial domain enhancement. In the formula, σ is the Sigmoid function, ⨂ represents element-wise multiplication, and G is the number of groups. Then on Perform upsampling or downsampling to make it consistent with Feature size alignment: Aligned and The concatenation process utilizes depthwise separable convolution (DWConv) with a linear transformation to learn the offset ∆ and the mask weights M at each position. in, This indicates the sampling position offset of each group of convolutional kernels. The mask weight is given at each position, and g is the number of groups. Deformable convolution dynamically adjusts the sampling position and weights of the convolution kernel by utilizing learned ∆ and M, as shown in the formula: in, Weight mask for the corresponding position For the k-th group of convolution kernels, This represents the offset of the k-th group at position p. Deformable convolution allows the convolution kernel to dynamically adjust the sampling position according to the target shape, thus better fitting the shape changes of distorted targets. Finally, the corrected Large-scale features Add them together to get the final output. It contains more complete information about distorted targets, making up for the network's shortcomings in feature fusion. The embodiments of the present invention disclosed above are merely illustrative of the invention. The embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-scale feature guidance module, characterized in that... In the deep semantic feature module, multi-scale features are adaptively fused through multiple receptive fields. Attention guidance is used to reduce semantic conflicts during feature fusion, thereby constructing a more effective global context model. The specific steps are as follows: The multi-scale feature guidance module takes features extracted from the backbone network as input. First, these input features undergo preliminary processing. Then, using dilated convolution operations with different dilation rates and combined with the residual structure, feature information with different scales is extracted. These multi-scale features are then concatenated and fused with the initial input features, and the result serves as the input for the latter part of the module. Here, Convs represents the combined use of 1x1 and 3x3 convolutions. This represents a dilated convolution with a 3x3 kernel and a dilation rate of i (i=1,3,5). The channel-dimensional weight distribution is constructed through parallel max pooling and average pooling operations; simultaneously, the Softmax function is used to generate an adaptive weight matrix for the spatial dimension. In the multi-scale feature guidance module, the concatenated and fused feature tensor is multiplied element-wise with the channel weights and spatial weights, and then weighted and superimposed to focus the features on key semantic regions, thereby extracting more refined global contextual features. Where ⨂ represents the dot product operation, and AP and MP represent the average pooling and max pooling operations, respectively.

2. A feature space correction module, characterized in that... The module can learn pixel offset values ​​based on context information and correct the position of the sampled features. The specific steps are as follows: The feature space correction module uses SGE attention to process the large-scale features of the input separately. and small-scale features Perform spatial domain enhancement. In the formula, σ is the Sigmoid function, ⨂ represents element-wise multiplication, and G is the number of groups. Then on Perform upsampling or downsampling to make it consistent with Feature size alignment: Aligned and The concatenation process utilizes depthwise separable convolution (DWConv) with a linear transformation to learn the offset ∆ and the mask weights M at each position. in, This indicates the sampling position offset of each group of convolutional kernels. The mask weight is given at each position, and g is the number of groups. Deformable convolution dynamically adjusts the sampling position and weights of the convolution kernel by utilizing learned ∆ and M, as shown in the formula: in, Weight mask for the corresponding position For the k-th group of convolution kernels, This represents the offset of the k-th group at position p. Deformable convolution allows the convolution kernel to dynamically adjust the sampling position according to the target shape, thus better fitting the shape changes of distorted targets. Finally, the corrected Large-scale features Add them together to get the final output. It contains more complete information about distorted targets, making up for the network's shortcomings in feature fusion.

3. A panoramic image feature enhancement target detection method, characterized by: A multi-scale feature guidance module is used to execute the multi-scale feature guidance operation steps described in claim 1 to construct a more effective global context model and improve the network's ability to perceive and represent target features at different scales. The feature space correction module is used to perform the feature space correction operation steps described in claim 2, correct the sampling feature position according to the context information, effectively capture the distorted target shape changes in multi-scale features, and improve the insufficient network feature fusion.

4. The panoramic image feature enhancement target detection method according to claim 3, characterized in that, The multi-scale feature guidance module and feature space correction module are plug-and-play modules that can be inserted into other target detection networks to enhance the panoramic image target detection capabilities of different networks.

Images