Open world target detection method

Abstract

The invention provides an open world target detection method, and belongs to the technical field of target detection. The method comprises the following steps: extracting multi-scale features of an input image through a deep convolutional neural network; inputting the multi-scale features into a cross-level-scale feature pyramid network to output fusion features; inputting the multi-scale features into a partial space co-occurrence module to output enhanced features; inputting the fused features and the enhanced features into a parallel adaptive feature fusion module, and outputting a shielding optimization feature map; inputting the shielding optimization characteristic pattern into a deformable encoder and a decoder to output a detection characteristic pattern; and obtaining a known category detection result and an unknown category mark based on the detection feature map. The method can effectively enhance the feature representation of the small target and the shielding target; when some parts of the target are seriously shielded, feature representation is improved through inter-part and intra-part co-occurrence, and the weight of important features in the model is improved; and the multi-scale adaptability and robustness of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to an open-world target detection method. Background Art

[0002] Object detection, a core task in computer vision, aims to accurately locate and identify objects of interest in images. In autonomous driving systems, object detection technology is a key component of environmental perception. By detecting objects such as vehicles, pedestrians, traffic signs, and obstacles in real time, it provides a reliable basis for system decision-making, thereby ensuring driving safety. Currently, the combination of deep learning and computer vision technologies has become the mainstream approach for open-world object detection. Relying on multimodal sensors such as cameras, lidar, and millimeter-wave radar, it enables the perception and analysis of complex road scenes.

[0003] Currently, open-world object detection primarily utilizes deep learning-driven methods. For example, Faster R-CNN utilizes a region proposal network to generate candidate boxes and combines classification and regression modules for object detection. Deformable DETR, on the other hand, utilizes a deformable attention mechanism to improve the model's adaptability to irregular objects. Furthermore, multi-sensor fusion (such as the complementary use of LiDAR and camera data) further enhances the model's robustness. These technologies have achieved high accuracy on standard test sets (such as COCO and KITTI), effectively handling the detection of clearly visible, medium-sized objects.

[0004] However, existing technologies still face significant challenges in complex open-world scenarios, especially in detecting small and occluded targets. Due to their low pixel count and lack of feature information, small targets are easily affected by background noise, resulting in a high missed detection rate. Occluded targets, on the other hand, lack some features, making it difficult for the model to extract effective representations, leading to false detections or positioning errors. These problems severely restrict the reliability of autonomous driving systems in real-world scenarios such as dense traffic and inclement weather. Although existing studies have optimized the system through multi-scale feature fusion or attention mechanisms, there is still a bottleneck of reduced detection accuracy under conditions of extreme scale changes and dynamic occlusion.

[0005] Therefore, there is an urgent need for an open-world object detection method that can improve the generalization ability of the model. Summary of the Invention

[0006] In view of this, the present invention provides an open-world target detection method, which effectively enhances the feature representation of small targets and occluded targets through a cross-level feature pyramid structure; improves the feature representation through inter-partial and intra-partial co-occurrence when some parts of the target are severely occluded through a partial spatial co-occurrence module, thereby increasing the weight of important features in the model; and focuses on important areas to enhance the multi-scale adaptability and robustness of the model.

[0007] To this end, the present invention provides the following technical solutions:

[0008] An open-world object detection method, comprising:

[0009] Extract multi-scale features of the input image through deep convolutional neural network;

[0010] Input the multi-scale features into a cross-level scale feature pyramid network to output fusion features;

[0011] Input the multi-scale features into a partial spatial co-occurrence module to output enhanced features;

[0012] The fused features and enhanced features are input into the parallel adaptive feature fusion module, and the occlusion optimized feature map is output;

[0013] The occlusion optimization feature map is input into the deformable encoder and the decoder outputs the detection feature map;

[0014] Based on the detection feature map, known category detection results and unknown category labels are obtained.

[0015] Furthermore, the cross-level feature pyramid network includes:

[0016] Feature selection module and feature fusion module;

[0017] The feature selection module includes: a global-local attention module and a dimension modulation module;

[0018] Feature screening is performed through the global-local attention module, and scale unification is performed through the dimension modulation module to output effective features;

[0019] The feature fusion module fuses the effective features and outputs a fusion feature.

[0020] Furthermore, the data processing flow of the feature fusion module includes:

[0021] Perform transposed convolution and bilinear interpolation on high-dimensional effective features to obtain pre-fused features;

[0022] That is, pre-fusion feature input channel attention, output fusion weight;

[0023] The low-dimensional effective features and pre-fusion features are weighted summed based on the fusion weight to obtain comprehensive features;

[0024] The comprehensive feature is concatenated with the pre-fusion feature to obtain the fusion feature.

[0025] Furthermore, the data processing process of the channel attention module includes:

[0026] The high-dimensional pre-fusion feature is processed by maximum pooling to obtain the first high-dimensional pre-fusion feature;

[0027] The high-dimensional pre-fusion features are processed by average pooling to obtain the second high-dimensional pre-fusion features;

[0028] The first high-dimensional pre-fused features and the second high-dimensional pre-fused features are concatenated and normalized to output a fusion weight.

[0029] Furthermore, the data processing process of the partial spatial co-occurrence module includes:

[0030] Generate candidate bounding boxes through a region proposal generation network based on multi-scale features;

[0031] Divide the candidate table frame into three parts: front, middle and back according to the preset value, and obtain the features of the front part, the middle part, the back part and the overall part;

[0032] Obtain the interest features corresponding to each part through interest pooling processing;

[0033] The adjacency matrix is ​​calculated based on each feature of interest, and enhanced features are generated through the graph convolution network.

[0034] Furthermore, the calculation process of the adjacency matrix includes:

[0035] Perform two branch processes on each feature of interest to obtain two reshaped features;

[0036] The branch processing includes: performing 1×1 convolution, ReLU activation function and reshaping processing in sequence;

[0037] Perform matrix multiplication and softmax calculation on the two reshaped features to output the adjacency matrix.

[0038] Furthermore, deep convolutional neural networks include: ResNet50 network.

[0039] Furthermore, the fused features and enhanced features are input into the parallel adaptive feature fusion module to output the occlusion optimized feature map, including:

[0040]

[0041] Among them, Attention srepresents spatially aware attention, L represents the number of layers in the pyramid, K is the number of sparsely sampled locations, and W l,k represents the kth convolution weight of layer l, X(l,p k +△P k ) is a feature Figure X In p k +△P k The eigenvalue of position p k Move to p by self-learning spatial offset k +△P k ,σ(△m k ) The self-learning importance weight calculated by the sigmoid function represents the position p k The attention weight of

[0042]

[0043] Among them, f(x) is a linear function approximated by a 1x1 convolutional layer; Represents the feature Figure X Perform global average pooling; Represents the feature Figure X Perform global maximum pooling, is the hard-sigmoid function.

[0044] The two-dimensional discrete cosine transform is used to compress the channel information and transform the input features Figure X Cut into multiple parts along the channel dimension [X0,X1,...,X n-1 ], where X i ∈R H×W×C′ , i∈{0,1,...,n-1}; for each X i Assign the corresponding two-dimensional discrete cosine transform frequency components, and the calculation formula is:

[0045]

[0046] Among them, Attention C represents channel-aware attention [u i ,V i ] is X i The corresponding frequency component 2D index; Freq i ∈R C′ The compressed C′-dimensional vector is then converted into a frequency domain representation Freq, which contains a variety of important frequency domain information of the feature map. Subsequently, the compression method is used to extract a variety of suitable frequency domain features to form a comprehensive vector Freq c ;

[0047] Use L2 norm to process the compressed frequency domain features, let s=S1,...,S c , perform normalization of the eigenvector, the formula is expressed as:

[0048]

[0049] Calculate the relative weight of the channel, formula:

[0050]

[0051] Among them, α is the weight parameter obtained through learning, and ε is a small constant for numerical stability;

[0052] By learning the gating weights of the channel γ=[γ1,...,γ c ] and gate bias β=[β1,...,β c ], adjust the degree of competition between this channel and other channels; attention weight S c and features Figure X Perform weighted multiplication, the formula is expressed as:

[0053] Y = Attention c (X)·X=[1+tanh(β+γS c )]·X

[0054] Where Y represents the occlusion optimized feature map.

[0055] Advantages and positive effects of the present invention:

[0056] The cross-level scale feature pyramid structure in the present invention can effectively enhance the feature representation of small targets and occluded targets through the feature fusion module. The feature fusion module helps to extract richer and more abstract semantic features by fusing high-dimensional feature maps with low-dimensional feature maps, thereby improving the detection capability of small targets and occluded targets; the partial space co-occurrence module can improve the feature representation through inter-partial and intra-partial co-occurrence when some parts of the target are severely occluded, thereby increasing the weight of important features in the model; the parallel adaptive feature fusion module improves the model performance through the space, scale and channel perception attention mechanism, focuses on important areas, enhances the perception of different scales, and improves the multi-scale adaptability and robustness of the model.

[0057] The global and local hybrid attention mechanism in the feature selection module of the method of the present invention can better capture relevant features in medium and long distances or global areas and relevant features in smaller ranges or local areas, and adaptively assign weights to each feature map through global pooling and fully connected layers.

[0058] The present invention uses the partial space co-occurrence module to cleverly divide a whole into three parts. The topological structure of the target is utilized through the graph convolutional network to enhance the features of the unobstructed part, thereby improving the model's detection effect on obstructed targets.

[0059] The present invention utilizes a parallel adaptive feature fusion module, which can effectively integrate the dynamic fusion of the model in both space and scale through parallel spatial perception attention and scale perception attention, thereby improving the performance and applicability of the model, and further optimizes the feature representation between different channels by adopting channel perception attention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0061] Figure 1 This is a framework diagram of an open-world object detection method according to an embodiment of the present invention;

[0062] Figure 2 This is a flow chart of the global-local attention module in an embodiment of the present invention;

[0063] Figure 3 This is a data processing flow chart of the global attention module in an embodiment of the present invention;

[0064] Figure 4 This is a data processing flow chart of the local attention module in an embodiment of the present invention;

[0065] Figure 5 This is a data processing flow chart of the feature fusion module in an embodiment of the present invention;

[0066] Figure 6 This is a data processing flow chart of the channel attention module in an embodiment of the present invention;

[0067] Figure 7 Schematic diagram of candidate area division in an embodiment of the present invention;

[0068] Figure 8 This is a schematic diagram of the convolutional network structure in the middle of an embodiment of the present invention;

[0069] Figure 9 This is a flow chart of adjacency matrix calculation in an embodiment of the present invention;

[0070] Figure 10 This is a data processing flow chart of the parallel adaptive feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0072] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0073] The present invention provides an open-world target detection method: a global and local hybrid attention module is used to filter the multi-scale features output by ResNet50 to output effective features; a feature fusion module is then used to fuse the effective features with high-dimensional features, and a graph convolutional neural network is used to capture the co-occurrence information between and within parts of different target parts to obtain enhanced features; the enhanced features are transferred together with the multi-scale feature map to a parallel adaptive feature fusion module, which obtains different receptive fields according to the size and shape of the target, dynamically allocates attention weights, and fuses features at different levels.

[0074] Example

[0075] Combine Figure 1 As shown in the flowchart, the open world target detection method in this embodiment includes the following steps:

[0076] S1, extract multi-scale features of the input image through deep convolutional neural network;

[0077] S2, input multi-scale features into the cross-level scale feature pyramid network to output fusion features;

[0078] S3, input the multi-scale features into the partial spatial co-occurrence module to output enhanced features;

[0079] S4, input the fused features and enhanced features into the parallel adaptive feature fusion module, and output the occlusion optimized feature map;

[0080] S5, inputting the occlusion optimization feature map into the deformable encoder and the decoder outputting the detection feature map;

[0081] S6. Obtain known category detection results and unknown category labels based on the detection feature map.

[0082] The specific steps of this method include:

[0083] S1. Extract multi-scale features of the input image through a deep convolutional neural network; in this embodiment, the deep convolutional neural network is a ResNet50 network, which obtains multi-scale features C1, C2, C3, C4 and C5.

[0084] S2, input multi-scale features into the cross-level scale feature pyramid network to output fusion features;

[0085] In this embodiment, C2, C3, C4, and C5 are input into a cross-level multi-scale feature pyramid network (CM-FPN). The cross-level multi-scale feature pyramid structure consists of a feature selection module and a feature fusion module.

[0086] 1) Feature selection module, including: global-local attention module and dimension modulation module;

[0087] ①Combination Figure 2 , feature screening is performed on the global-local attention module, and the output of valid features is further explained:

[0088] The feature map is input into the parallel global and local attention mechanisms, and the corresponding global attention feature map and local attention feature map are output; the global attention feature map and the local attention feature map are globally pooled, output to the fully connected layer for dimension conversion, and then fused, passed into the softmax to obtain the weight, and the global attention feature map and the local attention feature map are further weighted to obtain the attention fusion feature.

[0089] The global attention module consists of a channel attention submodule and a spatial attention submodule; it is a data processing module. Figure 3 As shown;

[0090] The channel attention submodule uses 3D permutation to retain information across three dimensions, and then uses two layers of MLP to amplify the channel spatial dependencies across dimensions. After reverse permutation, it is normalized using the sigmoid function and weighted with the original feature map.

[0091] The spatial attention submodule uses two 7x7 convolutional layers in series to fuse spatial information, and then uses the sigmoid function to normalize and weight the feature map.

[0092] Local attention module, the structure is as follows Figure 4 As shown in the figure, it includes two branches with different receptive fields, one is two series-connected Conv+BN+ReLU, and the other is a single Conv+BN+ReLU. The feature maps output by the two branches are then concatenated and passed into the global pooling. They are then input into FC and softmax to obtain weight values, and the feature maps output by the two branches are weighted, and finally element-wise addition is performed.

[0093] ② The dimension modulation module reduces the number of channels of the feature map of each scale to 256 through 1×1 convolution.

[0094] In the dimension modulation module, 1x1 convolution is applied to reduce the number of channels of the feature map of each scale to 256, and feature maps of four dimensions P2, P3, P4 and P5 are obtained.

[0095] 2) The feature fusion module uses high-dimensional features as weights to filter the basic semantic information in low-dimensional features, thereby fusing features; the data processing flow of the feature fusion module is as follows: Figure 5 As shown;

[0096] The data processing flow of the feature fusion module includes: performing transposed convolution and bilinear interpolation on high-dimensional effective features to obtain pre-fused features; inputting the pre-fused features into the channel attention (CA) and outputting the fusion weights; performing weighted summation of low-dimensional effective features and pre-fused features based on the fusion weights to obtain comprehensive features; and concatenating the comprehensive features with the pre-fused features to obtain fused features. Specifically:

[0097] 1) An input high-dimensional feature F5(F high ∈R C×H×W ) and an input low-dimensional feature Fi(F low ∈R C ×H1×W1 ) High-dimensional features are expanded using transposed convolution with a step size of 2 and a convolution kernel size of 3x3, resulting in a feature size of F high ∈R C×2H×2W ; Then, to unify the dimensions of high-dimensional and low-dimensional features, bilinear interpolation is used to upsample and downsample the high-dimensional features to obtain the pre-fused features F att ∈R C×H1×W1 ;

[0098] 2) Input the pre-fused features into the channel attention module. The channel attention module structure is as follows Figure 6 As shown, the output fusion weights are:

[0099] The pre-fusion feature is processed by maximum pooling to obtain a first pre-fusion feature; the pre-fusion feature is processed by average pooling to obtain a second pre-fusion feature; the first pre-fusion feature and the second pre-fusion feature are spliced ​​and normalized to output the fusion weight.

[0100] 3) Fuse the pre-fused features with the weighted low-dimensional features to enhance the feature representation of the model. The formula is:

[0101] F att =BL(T-Conv(F high )

[0102] F out =F low *CA(F att )+F att

[0103] Among them, CA represents channel attention; F out Indicates; F high represents the pre-fusion feature; F low Represents low-frequency features.

[0104] In this embodiment, the multi-scale feature map is subjected to the feature fusion module to obtain feature maps of four dimensions: F2, F3, F4, and P5, where F2, F3, and F4 are obtained by fusing the features of P5, P2, P3, and P4, respectively. Subsequently, the feature maps of the four dimensions are subjected to 1x1 convolution to convert the number of channels of the feature maps to 256.

[0105] S3, input the multi-scale features into the partial spatial co-occurrence module (PSC) to output enhanced features;

[0106] In the driving field, targets usually have fixed proportions and limited deformation, and are suitable for division using a fixed empirical proportion. In order to utilize the spatial co-occurrence information of different target parts as clues to the occluded parts, a partial spatial co-occurrence module is designed to capture the spatial co-occurrence relationship of target parts through a graph convolutional neural network.

[0107] The Partial Spatial Co-occurrence Module combines the object topology with a graph convolutional network to capture both intra-part and inter-part co-occurrence information (inter-part refers to the relative relationships and dependencies between different locations, while intra-part refers to the feature composition and internal dependencies of each location). This co-occurrence information helps improve feature representation under varying degrees of occlusion, especially when parts of the object are severely occluded.

[0108] The data processing process of the partial spatial co-occurrence module includes:

[0109] 1) Generate candidate bounding boxes through a region proposal generation network based on multi-scale features;

[0110] The region proposal network receives the feature map C5 as input and uses RPN (Region Proposal Network) to generate candidate regions through sliding windows and anchor points.

[0111] 2) In order to deal with the occlusion problem, the candidate frame is divided into three parts: front, middle and back according to the preset value, and the front part features, middle part features, back part features and overall part features are obtained; (front: Fbefore, middle: Fmiddle, back: Fbehind); Figure 7 shown.

[0112] 3) Obtain the interest features corresponding to each part through interest pooling processing;

[0113] The RoIPool (Region of Interest Pooling) operation is performed on each part and the whole to generate four RoI pooling features, and the number of channels of each part is compressed to 128 through an additional 1x1 convolution, and the 512-channel interest feature representation is obtained by merging.

[0114] 4) Calculate the adjacency matrix based on each feature of interest and generate enhanced features through the graph convolution network.

[0115] Their interesting feature representation is enhanced by considering the intra-part co-occurrence of each target part.

[0116] In this example, the rear part of a car is blocked, which makes some features of the rear part invisible. The graph convolutional network is used to capture the interesting features F of the target part. m The spatial relationship between different sub-regions (the taillights and rear windows) in the image.

[0117] ① Interest characteristics F m First, the convolution operation is performed to obtain the partial inner space adjacency matrix A s .

[0118] Perform two branch processes on each feature of interest to obtain two reshaped features; the branch processes include: performing 1×1 convolution, ReLU activation function and reshaping in sequence; performing matrix multiplication and softmax calculation on the two reshaped features to output the adjacency matrix. Specifically:

[0119] Interest feature F m Through two parallel 1x1 convolution layers; then processed by ReLU activation function, two feature maps are obtained. Next, these two feature maps are reshaped, and then matrix multiplication is performed, and finally the spatial adjacency matrix A is calculated through softmax operation.s ; Then use graph convolution to capture the spatial relationship between different sub-regions (the taillights and rear windows) in the feature representation of interest. The formula is expressed as:

[0120]

[0121] Among them, σ is the ReLU activation function; W s ∈R C×C is the learnable parameter matrix; C is the number of channels, which is 128 in this embodiment; A s ∈R H×W×H×W It is Figure G s =(V s ,A s ) is used to represent the spatial relationship between each pixel in the RoI area; V s Represents the pixel nodes within the area; It is the feature after graph convolution, which takes the spatial relationship into consideration.

[0122] ② Enhance the features output by the graph convolution layer With input feature F m Add together to get enhanced features.

[0123] ③ By introducing a graph convolution layer, we can capture the relationship between different parts of the target, including the entire object, to deal with the situation where a certain part is severely occluded. Based on the partial enhancement features, we obtain enhanced features through a graph convolutional network.

[0124] Define the graph structure, graph G p =(V p ,A p ), each part (three parts plus the whole) is a separate node V in the graph p ; A p is the spatial adjacency matrix of the graph, capturing the relationships between different nodes.

[0125] Graph convolution operation uses graph convolution layer to capture the relationship between different parts and the whole. The formula is expressed as:

[0126]

[0127] Among them, F e ∈R n×d is the enhanced feature of n parts; A p is the spatial adjacency matrix; W p ∈R d×d is a learnable parameter matrix; node features are propagated through Laplace smoothing (IA p ).

[0128] During severe occlusion, the enhanced partial features of a specific node may not contain relevant partial information. Therefore, it is desirable to give A p To this end, a self-attention scheme is introduced on each edge to assign a learnable weight a ij . Introduce the self-attention mechanism to each edge and assign a learnable weight a to each edge ij , self-attention is calculated by combining the features of node i and node j through full connection and sigmoid activation, and the formula is expressed as:

[0129] a ij =sigmoid(W T ·(F i ||F j ))

[0130] Among them, W T is the learnable weight matrix; F i,j is the feature of node i, j, and || represents the concatenation operation.

[0131] Unnormalized spatial adjacency matrix definition:

[0132]

[0133] Calculate the adjacency matrix using L2 regularization Get an A p (i,j).

[0134] S4, input the fused features and enhanced features into the Parallel Adaptive Feature Fusion (PAFF) module, and output the occlusion optimized feature map;

[0135] Spatial-aware attention is used to sparsify attention through deformable convolution and self-learned spatial offsets, focusing on specific spatial locations and reducing the spatial bias between upper and lower spatial features before fusion. Scale-aware attention is also used to dynamically fuse features across different scales, enhancing the model's ability to perceive objects of varying scales. This dynamic fusion of spatial and scale aspects improves model performance and adaptability. Finally, perceptual attention is used to enable interaction between features through frequency domain analysis and a gating mechanism. The channel-aware attention module further weights the fused feature maps, combining frequency domain information with a gating mechanism to optimize feature representation across different channels, improving model performance and adaptability.

[0136] 1) Spatial-aware attention: To handle the correlation between geometric transformations of different object shapes and features at different spatial locations, we introduce spatial-aware attention. This focuses on discovering discriminative regions between different spatial locations and feature layers, reducing the bias between upper and lower spatial features and ensuring a more reasonable allocation of spatial attention weights for subsequent dynamic fusion of spatial features. Considering the high-dimensional feature representation, this module is divided into two steps:

[0137] First, a deformable convolution operation is used to calculate the self-learning spatial offset, so that the attention mechanism can adaptively focus on local areas based on the content of the input features. By sparsely focusing the attention mechanism, only specific spatial locations are focused, so that attention is concentrated on the most relevant spatial areas, reducing spatial deviation.

[0138] Secondly, cross-level feature aggregation is performed on these same spatial locations to promote information fusion and sharing. This method can improve spatial perception and help accurately identify specific spatial location features. The specific calculation formula is:

[0139]

[0140] Where L is the number of layers in the pyramid, K is the number of sparse sampling locations, and W is the number of l,k represents the kth convolution weight of layer l, X(l,p k +△P k ) is a feature Figure X In p k +△P k The eigenvalue of position p k Move to p by self-learning spatial offset (offset) k +△P k , such movement is to focus more on the discrimination area, σ(△m k ) The self-learning importance weight calculated by the sigmoid function represents the position p k The attention weight of .

[0141] 2) Scale-aware attention: Targeting the scale differences of objects and the correlation between features at different levels, we introduce a scale-aware attention method to improve feature learning at different levels and enhance the model's ability to perceive objects of different scales. This method assigns weights based on semantic importance to ensure accurate perception of different scales and dynamically fuse features of different scales. The calculation formula is:

[0142]

[0143] Among them, f(x) is a linear function approximated by a 1x1 convolutional layer; Represents the feature Figure X Perform global average pooling; Represents the feature Figure X Perform global maximum pooling, is the hard-sigmoid function.

[0144] 3) Channel-aware attention C :The representation of different targets may be related to the characteristics of different channels. The channel attention method is introduced to improve the representation learning of different channel features and enhance the model's perception of channel information.

[0145] First, the two-dimensional discrete cosine transform (2DDCT) is used to compress the channel information and transform the input features into Figure X Cut into multiple parts along the channel dimension [X0,X1,...,X n-1 ], where X i ∈R H×W×C′ , i∈{0,1,...,n-1}. For each X i Assign the corresponding two-dimensional discrete cosine transform frequency components, and the calculation formula is:

[0146]

[0147] Among them, [u i ,v i ] is X i The corresponding frequency component 2D index; Freq i ∈R C′ The compressed C′-dimensional vector is then converted into a frequency domain representation Freq, which contains a variety of important frequency domain information of the feature map. Subsequently, the compression method is used to extract a variety of suitable frequency domain features to form a comprehensive vector Freq c .

[0148] Secondly, the compressed frequency domain features are processed using the L2 paradigm to create a competitive relationship between channels. Let s=S1,...,S c , perform normalization of the eigenvector, the formula is expressed as:

[0149]

[0150] Calculate the relative weight of the channel, formula:

[0151]

[0152] Among them, α is the weight parameter obtained through learning, and ε is a small constant for numerical stability.

[0153] Finally, a gating mechanism is introduced to promote channel competition and cooperation.

[0154] By learning the gating weights of the channel γ=[γ1,...,γ c ] and gate bias β=[β1,...,β c ], adjust the degree of competition between this channel and other channels, that is, Attention c (X). When the channel's gating power γ c When activated, Attention c Promote the channel to compete with other channels. When negatively activated, Attention c Encourage this channel to cooperate with other channels. Attention weight S c and features Figure X Perform weighted multiplication, the formula is expressed as:

[0155] Y = Attention c (X)·X=[1+tanh(β+γS c )]·X

[0156] Among them, γ and β are initialized to 0; 1×1 convolution is used to convert the number of channels of the feature map to 256; and the occlusion optimized feature map Y is obtained.

[0157] S5, inputting the occlusion optimization feature map into the deformable encoder and the decoder outputting the detection feature map;

[0158] The occlusion-optimized feature map is fed into the deformable encoder, along with the position code and reference points. The position code is used to add position information so the model understands the location of the feature, and the reference points are used to guide the attention mechanism to focus on a specific location. The deformable attention module receives the feature map, position code, and reference points, dynamically samples feature maps of different resolutions through learning offsets, calculates attention, and then passes it to Add&Norm to perform a residual connection between the input features and the attention output. This is then normalized and passed to the feedforward network, which includes two fully connected layers and a ReLU activation function to further process the normalized features. Add&Norm is used again to perform a residual connection between the input features and the feedforward network, followed by normalization. The processed feature map is passed to the Deformable Decoder.

[0159] Output detection feature map through deformable decoder;

[0160] 1) First, the target query is input into the multi-head self-attention, which is equivalent to the target representation that the model needs to detect. The key and value are features passed from the encoder. The self-attention mechanism calculates the similarity score (attention weight) between the query and the key, and uses these weights to weighted sum the values ​​to obtain a new feature representation. The updated query is output, which contains the global context information extracted by the self-attention mechanism.

[0161] 2) Add&Norm is passed in to perform residual connection between the input query and the output of the multi-head self-attention mechanism, and perform layer normalization to stabilize and accelerate the training process.

[0162] The query output by the Add&Norm layer is then passed into MSDeformAtten. By learning the offset, the multi-scale feature map is dynamically sampled and the attention weight is calculated to obtain a feature representation with multi-scale information. The weighted feature representation is output, which integrates information at different scales and improves the model's perception of the target position and shape.

[0163] 3) The data is then passed to Add&Norm, which performs a residual connection between the input and the output of the multi-scale deformable attention mechanism and layer normalization. The output is then passed to the FNN (Feedforward Neural Network), which performs a nonlinear transformation on the normalized features to enhance the expressive power of the feature representation.

[0164] 4) Finally, Add&Norm is passed in to perform residual connection on the input features and the output of the feedforward network, perform layer normalization, and obtain the final target features.

[0165] S6. Obtain known category detection results and unknown category labels based on the detection feature map.

[0166] 1) Extract embeddings representing different targets from the final target features output by the deformable decoder, use the Hungarian matching algorithm to perform target matching, and determine the best matching pair.

[0167] 2) Output the best matching pair to f obj (target existence characteristics), f cls (target classification feature) and f bbox (bounding box regression features);

[0168] f obj The Mahalanobis distance and distribution parameters are used to estimate the matching object embedding, distinguish the feature distribution of matched and unmatched targets, and calculate the difference in target embedding to generate the loss (L o), the similarity between targets is measured by calculating the Mahalanobis distance of the target embedding, and the target existence score is finally output. The target query embedding is input into f cls and f bbox In , the predicted score and bounding box parameters of the target are output respectively;

[0169] f cls Sigmoid loss is used for training, f bbox L1 and GIoU losses are used for training.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open world object detection method, characterized in that: include: Extract multi-scale features of the input image through deep convolutional neural network; Input the multi-scale features into a cross-level scale feature pyramid network to output fusion features; Input the multi-scale features into a partial spatial co-occurrence module to output enhanced features; The fused features and enhanced features are input into the parallel adaptive feature fusion module, and the occlusion optimized feature map is output; The occlusion optimization feature map is input into the deformable encoder and the decoder outputs the detection feature map; Based on the detection feature map, known category detection results and unknown category labels are obtained.

2. The method according to claim 1, characterized in that The cross-level feature pyramid network includes: Feature selection module and feature fusion module; The feature selection module includes: a global-local attention module and a dimension modulation module; Feature screening is performed through the global-local attention module, and scale unification is performed through the dimension modulation module to output effective features; The feature fusion module fuses the effective features and outputs a fusion feature.

3. The method according to claim 2, characterized in that The data processing flow of the feature fusion module includes: Perform transposed convolution and bilinear interpolation on high-dimensional effective features to obtain pre-fused features; That is, pre-fusion feature input channel attention, output fusion weight; The low-dimensional effective features and pre-fusion features are weighted summed based on the fusion weight to obtain comprehensive features; The comprehensive feature is concatenated with the pre-fusion feature to obtain the fusion feature.

4. The method according to claim 3, characterized in that The data processing process of the channel attention module includes: The high-dimensional pre-fusion feature is processed by maximum pooling to obtain the first high-dimensional pre-fusion feature; The high-dimensional pre-fusion features are processed by average pooling to obtain the second high-dimensional pre-fusion features; The first high-dimensional pre-fused features and the second high-dimensional pre-fused features are concatenated and normalized to output a fusion weight.

5. The method according to claim 1, wherein The data processing process of the partial space co-occurrence module includes: Generate candidate bounding boxes through a region proposal generation network based on multi-scale features; Divide the candidate table frame into three parts: front, middle and back according to the preset value, and obtain the features of the front part, the middle part, the back part and the overall part; Obtain the interest features corresponding to each part through interest pooling processing; The adjacency matrix is ​​calculated based on each feature of interest, and enhanced features are generated through the graph convolution network.

6. The method according to claim 5, characterized in that The calculation process of the adjacency matrix includes: Perform two branch processes on each feature of interest to obtain two reshaped features; The branch processing includes: performing 1×1 convolution, ReLU activation function and reshaping processing in sequence; Perform matrix multiplication and softmax calculation on the two reshaped features to output the adjacency matrix.

7. The method according to claim 1, characterized in that The deep convolutional neural network includes: a ResNet50 network.

8. The method according to claim 1, characterized in that The step of inputting the fused features and the enhanced features into a parallel adaptive feature fusion module and outputting an occlusion optimized feature map comprises: Among them, Attention s represents spatially aware attention, L represents the number of layers in the pyramid, K is the number of sparsely sampled locations, and W l,k represents the kth convolution weight of layer l, X(l,p k +△P k ) is the feature map X in p k +△P k The eigenvalue of position p k Move to p by self-learning spatial offset k +△P k ,σ(△m k ) The self-learning importance weight calculated by the sigmoid function represents the position p k The attention weight of Among them, f(x) is a linear function approximated by a 1x1 convolutional layer; Indicates global average pooling of feature map X; Indicates the global maximum pooling of the feature map X, is the hard-sigmoid function. The two-dimensional discrete cosine transform is used to compress the channel information and cut the input feature map X into multiple parts along the channel dimension [X0, X1, ..., X n-1 ], where X i ∈R H×W×C′ , For each X i Assign the corresponding two-dimensional discrete cosine transform frequency components, and the calculation formula is: Among them, Attention C represents channel-aware attention [u i ,v i ] is X i The corresponding frequency component 2D index; Freq i ∈R C′ The compressed C′-dimensional vector is then converted into a frequency domain representation Freq, which contains a variety of important frequency domain information of the feature map. Subsequently, the compression method is used to extract a variety of suitable frequency domain features to form a comprehensive vector Freq c ; Use L2 norm to process the compressed frequency domain features, let s=S1,...,S c , perform normalization of the eigenvector, the formula is expressed as: Calculate the relative weight of the channel, formula: Among them, α is the weight parameter obtained through learning, and ε is a small constant for numerical stability; By learning the gating weights of the channel γ=[γ1,...,γ c ] and gate bias β=[β1,...,β c ], adjust the degree of competition between this channel and other channels; attention weight S c Perform weighted multiplication with the feature map X, and the formula is expressed as: Y=Attention c (X)·X=[1+tanh(β+γS c )]·X Where Y represents the occlusion optimized feature map.

Images