Open-world target detection method based on progressive feature fusion and uncertainty estimation

Abstract

This invention discloses an open-world object detection method based on progressive feature fusion and uncertainty estimation. It uses an open-world object detection model to detect objects in natural images, determining the coordinates and confidence scores of known and unknown classes. The open-world object detection model includes: acquiring an open-world object detection dataset and performing data preprocessing; constructing an open-world object detection model using Fast R-CNN and extracting initial multi-scale features using ResNet-50; progressively fusing the initial multi-scale features to obtain fused multi-scale features; performing attention calculations on the fused multi-scale features to obtain fixed features; predicting the classification score of the fixed features; calculating the variance of the classification score for uncertainty estimation; determining whether the class belongs to a known or unknown class based on the uncertainty and the classification score; performing backdoor adjustments based on the uncertainty to obtain the final classification score; and training the open-world object detection model using the open-world object detection dataset.

Description

Technical Field

[0001] This invention relates to an open-world target detection method based on progressive feature fusion and uncertainty estimation, belonging to the field of image processing technology. Background Technology

[0002] With the rapid development of deep learning technology, it is being applied to an increasing number of fields. In the field of natural image processing, there is a significant demand for such processing, leading to the widespread application of deep learning-based computer vision techniques in various tasks. Open-world object detection was proposed to address the limitations of traditional object detection methods when faced with new environments and objects. In traditional object detection, all categories to be detected are labeled during the training phase. Open-world object detection, however, relaxes this prerequisite, allowing the model to encounter natural images of unknown categories during the testing phase and to detect and learn from these unknown categories.

[0003] Open-world object detection is a fundamental task in computer vision. Its main objective is to scan natural images for objects that may need to be identified, find all known and possible unknown categories, and label the location and category of each object. The model framework for open-world object detection needs to recognize objects of known categories and gradually adapt to unknown categories, utilizing techniques such as incremental learning and self-supervised learning to improve detection capabilities in continuously expanding dynamic environments.

[0004] In traditional closed-world environments, existing object detection frameworks are relatively well-developed. However, applying these frameworks to open-world environments still presents several challenges. Firstly, the lack of effective supervision for unknown categories during training can cause models to incorrectly identify known categories as unknown ones. Secondly, as new categories are continuously introduced, models are prone to forgetting previously learned knowledge. Furthermore, open-world environments require models to autonomously discover and learn new categories, thereby reducing reliance on extensive manual annotation. The challenges of open-world object detection are as follows:

[0005] (1) Confusion between known and unknown categories: Due to limitations in training data, feature similarity, and the lack of a mechanism to identify "unknown" categories, the model may lack accuracy in recognizing new categories. This not only affects detection performance but also reduces the model's robustness and decision reliability in real-world scenarios.

[0006] (2) Low recall for unknown categories: Since the model is usually trained only on known categories, its recognition and detection capabilities are limited when encountering unknown categories, resulting in the inability to effectively identify all new category objects. This low recall problem stems from the model's lack of sensitivity to the features of unknown categories, especially when there are significant differences in feature distribution compared to known categories. The model struggles to identify these objects as independent new categories, thus affecting the comprehensiveness and applicability of the detection.

[0007] (3) Catastrophic forgetting: In open-world environments, models need to constantly adapt to new categories. However, the incremental learning process lacks repeated training on existing categories, causing the model's recognition accuracy to drop significantly when new categories are added. This phenomenon leads to the model losing its ability to recognize existing categories while expanding to new categories, affecting the continuity and accuracy of detection. Summary of the Invention

[0008] The technical problem to be solved by this invention is to overcome the defects of the prior art and provide an open-world object detection method based on progressive feature fusion and uncertainty estimation. It also proposes corresponding solutions to the problems of confusion between known and unknown classes and low recall of unknown classes.

[0009] Preferably, the present invention provides an open-world target detection method based on progressive feature fusion and uncertainty estimation, characterized in that it includes:

[0010] The trained open-world object detection model is used to detect natural images of natural scenes, and to determine the classification results of known category features, the location information of known category features, the target score of known category features, the classification results of unknown category features, the location information of unknown category features, and the target score of unknown category features.

[0011] The trained open-world object detection model includes:

[0012] Step 1: Obtain the open-world object detection dataset and preprocess the data in the open-world object detection dataset;

[0013] Step 2: Construct an open-world object detection model using an improved Fast R-CNN network. The open-world object detection model uses ResNet-50 as the backbone network to extract initial multi-scale feature maps; perform weighted progressive fusion on the initial multi-scale feature maps to obtain fused multi-scale feature maps; perform cross-attention calculation on the fused multi-scale feature maps to obtain fixed-size feature maps; and calculate category classification scores, bounding box regression scores, and object detection scores on the fixed-size feature maps.

[0014] Step 3: Estimate the uncertainty of the category classification score, calculate the variance of the category classification score using Monte Carlo sampling as the confidence level of the classification result, and quantify the uncertainty based on this confidence level; determine whether the category corresponding to the category classification score belongs to a known category or an unknown category based on the uncertainty; and obtain the final category classification score by backdoor adjustment based on the uncertainty.

[0015] Step 4: Train the open-world object detection model using the open-world object detection dataset.

[0016] Firstly, in step 1, obtain the open-world object detection dataset and preprocess the data in the open-world object detection dataset, including:

[0017] Step (1.1) Obtain the open-world object detection dataset. The open-world object detection dataset contains natural images with labeled object instances. The open-world object detection dataset has multiple known categories and several unknown categories.

[0018] Step (1.2) involves cropping the natural images in the open-world object detection dataset to obtain images of a preset size; and modifying the instance information in the open-world object detection dataset to fit the images of the preset size.

[0019] Step (1.3) involves marking the image of the preset size, re-dividing it into several tasks, and performing incremental learning on the tasks.

[0020] Prior to this, step 2 includes:

[0021] Step (2.1) defines the feature maps of the backbone network ResNet50 at different scales as {C1,C2,C3,C4,C5}, where C1 corresponds to the features of the initial convolutional layer and pooling layer, and C2,C3,C4,C5 correspond to the features of the subsequent layers.

[0022] In step (2.2), C1 is substituted into a convolutional layer, a batch normalization layer, and a ReLU activation function for processing to obtain C2;

[0023] Substituting C2 into a convolutional layer, a batch normalization layer, and a ReLU activation function, we obtain C3;

[0024] C3 is processed by substituting it into a convolutional layer, a batch normalization layer, and a ReLU activation function to obtain C4;

[0025] C4 is processed by substituting it into a convolutional layer, a batch normalization layer, and a ReLU activation function to obtain C5;

[0026] In step (2.3), global max pooling is applied to C1, reducing the width and height of C1 to 1 / 4 of their original values. Convolution with stride of 1 and 1x1 is applied to C2 to obtain the fused feature map P2.

[0027] Step (2.4) applies a 1x1 convolution and bilinear interpolation to C2 for upsampling, so that the spatial dimensions of C2 are aligned with those of C3, resulting in C2′; and applies a 2x2 convolution with a stride of 2 to C3, so that C3 is aligned with the feature dimensions of C2′.

[0028] Step (2.5): Calculate the weights of C2′ and C3 and perform weighted fusion to obtain the fused feature map P3;

[0029] Step (2.6): Repeat steps (2.4) and (2.5) above, and weightedly fuse the features of C3 and C4, and the features of C4 and C5. When weightedly fusing the features of C3 and C4, a convolution with a stride of 4 and a stride of 4x4 is applied to obtain the fused feature map P4. When weightedly fusing the features of C4 and C5, a convolution with a stride of 8 and a stride of 8x8 is applied to obtain the fused feature map P5. A set of fused multi-scale feature maps {P2, P3, P4, P5} is obtained.

[0030] Step (2.7) involves performing cross-attention calculation on the fused multi-scale feature maps to obtain a fixed-size feature map; then, the fixed-size feature map is scored for category classification, bounding box regression, and target identification.

[0031] Preferred, step (2.6) includes:

[0032] For each pair of data in the feature map {(C1,C2),(C2,C3),(C3,C4),(C4,C5)}, perform the following steps:

[0033] Step (2.41) involves applying a convolutional layer, a batch normalization layer, and a ReLU activation function to a pair of feature maps to obtain the first feature and the second feature.

[0034] Step (2.42) involves concatenating the first and second features along the channel dimension to obtain the third feature map;

[0035] Step (2.43) involves applying a 1x1 convolution to the third feature map to calculate the weights of each level of feature map.

[0036] Step (2.44) uses the softmax function to normalize the weights of each feature map to obtain the fusion weights of each feature map;

[0037] Step (2.45): Based on the fusion weight of each feature map, the feature maps of different scales are weighted and fused to obtain the fused feature map;

[0038] Step (2.46) applies a convolutional layer, a batch normalization layer and a ReLU activation function to the fused feature map to obtain the final weighted fused feature map;

[0039] Step (2.47) combines all the final weighted fusion feature maps to obtain the final weighted multi-scale fusion feature map {P2,P3,P4,P5}.

[0040] Prioritize, in step 3, perform uncertainty estimation on the category classification scores, calculate the variance of the category classification scores using Monte Carlo sampling as the confidence level of the classification result, and quantify the uncertainty based on this confidence level; determine whether the category corresponding to the category classification score belongs to a known category or an unknown category based on the uncertainty; perform backdoor adjustment based on the uncertainty to obtain the final category classification score, including:

[0041] Step (3.1) involves fusing the feature map F from the i-th sampling. i Each pixel in the array undergoes N forward propagation iterations, where i = 1, 2, ..., N;

[0042] Step (3.2) involves processing the fused feature map F obtained from the i-th sampling. i Classify using classifier C to obtain the predicted probability distribution P for each category. i (c);

[0043] Step (3.3) calculates the mean μ(c) and variance σ of the predicted probability for each category c. 2 (c) , where uncertainty is obtained;

[0044]

[0045] Step (3.4): Based on the uncertainty, evaluate the classification score of each natural image to determine whether there is an unknown category; if the classification score of a natural image exceeds the first preset unknown category threshold, it is determined that there is an unknown category.

[0046] Step (3.5) uses the calculated uncertainty as a measure of uncertainty in the backdoor adjustment.

[0047] Preferred, step (3) includes:

[0048] Step (3.51) involves performing N forward propagations on the category classification scores during the inference process of the open-world object detection model;

[0049] Step (3.52): For each category c, calculate the average probability value u(c) of all N Monte Carlo samplings;

[0050] Step (3.53): For each class c, calculate the variance σ of the predicted probabilities from all N Monte Carlo samplings. 2 (c);

[0051] Step (3.54), using u(c) and σ 2 (c) Evaluate the prediction uncertainty of each natural image in the open world object detection dataset for a specific category. If the classification score of the natural image exceeds the second preset unknown category threshold, it is determined that there is an unknown category.

[0052] Step (3.55) involves calculating the variance σ. 2 (c) As a confidence uncertainty measure for each natural image in the open-world object detection dataset, to dynamically adjust the classification score of each natural image.

[0053] Preferred, step (3.5) includes:

[0054] (3.51) If the variance σ 2 (c) If the value exceeds the preset threshold τ, the corresponding natural image is determined to have high uncertainty;

[0055] (3.52) The variance value σ of each high-uncertainty natural image 2 (c) As an adjustment factor, it is added to the classification score of natural images;

[0056] (3.53) Calculate the classification score S′ of the natural image for each high-uncertainty sample:

[0057] S′=S+β·σ 2 (c),

[0058] Where S is the original classification score, and β is a preset coefficient;

[0059] (3.54) If the backdoor adjusted score S′ of each category c exceeds the adjusted third classification threshold, then the corresponding natural image is determined to belong to category c.

[0060] Prior to step (2.7), cross-attention calculation is performed on the fused multi-scale feature maps to obtain a fixed-size feature map, including:

[0061] (2.71) Receive input feature map f, the dimension of feature map f is (b,n,c), b is the batch size, n is the number of detection boxes, and c is the feature dimension;

[0062] (2.72), the feature dimension dim = 64 and the input resolution of feature map f is 1024;

[0063] (2.73) The feature map f is processed by depthwise separable convolution using the nn.Conv1d(dim,dim,3,padding=1,groups=dim) convolutional layer to obtain the processed feature map; the processed feature map is added to the original input feature map to form a residual connection;

[0064] (2.74) Normalize the input feature map, then use a linear layer to transform the feature map and apply the activation function, repeat this step twice;

[0065] (2.75) Based on the input feature map obtained in step (2.74), calculate the cross attention to obtain the weighted feature representation;

[0066] (2.75) The weighted feature representation is transformed by a linear layer, and the weighted feature representation is added to the residual connection to obtain the feature map y;

[0067] (2.76) The feature map y is processed by depthwise separable convolution using the nn.Conv1d(dim,dim,3,padding=1,groups=dim) convolutional layer;

[0068] (2.77) The input feature map output from step (2.76) is normalized and further processed by the MLP layer to generate a fixed-size output feature map.

[0069] Preferably, step (2.75) includes:

[0070] The dimensions of the acquired input feature map x are expanded into (b, n, num_heads, head_dim), where b is the batch size, n is the number of detection boxes, num_heads is the number of attention heads, and head_dim is the dimension of the attention head.

[0071] Calculate self-attention for x:

[0072] Q=φ(xW Q ), (1),

[0073] K=φ(xW K (2),

[0074] V=φ(xW V (3),

[0075]

[0076] Attn normed =Norm(Attn)+x, (5),

[0077] output = Norm(MLP(Attn) normed ))+Attn normed (6),

[0078] Among them, W Q W K W V These are the weight parameters, Q, K, and V are variables in the calculation process, Norm represents the regularization function, MLP is a multilayer perceptron, and output is the final weighted feature representation.

[0079] Preferably, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in any of the first aspects.

[0080] Preferably, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0081] The beneficial effects achieved by this invention are as follows:

[0082] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0083] First, because conventional techniques lack effective supervision for unknown categories during training, the features of unknown categories inevitably depend on the features of known categories. This poses a significant challenge to the distinction between known and unknown categories, leading many open-world object detection models to produce spurious relevance when handling unknown categories. This invention, through progressive feature fusion and attention mechanisms, enables the model to effectively fuse feature maps of different scales, enhancing feature representation capabilities and improving the detection accuracy for known categories.

[0084] Second, this invention proposes an uncertainty estimation module that can adjust the scores after classification to a certain extent. This method calculates the variance and mean through Monte Carlo sampling to obtain the probability of the unknown category's existence, and then applies the variance to adjust the score of the unknown category. This method can effectively improve the recall rate of the unknown category while maintaining the detection accuracy of the known category. Attached Figure Description

[0085] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a flowchart of the process of this invention;

[0087] Figure 2 This is an overall structural diagram of the network of the present invention;

[0088] Figure 3 This is a schematic diagram of the MLLABlock attention network structure. Detailed Implementation

[0089] See Figure 1 This application discloses an open-world target detection method based on progressive feature fusion and uncertainty estimation, characterized by comprising:

[0090] The trained open-world object detection model is used to detect natural images of natural scenes, and to determine the classification results of known category features, the location information of known category features, the target score of known category features, the classification results of unknown category features, the location information of unknown category features, and the target score of unknown category features.

[0091] The trained open-world object detection model includes:

[0092] Step 1: Obtain the open-world object detection dataset and preprocess the data in the open-world object detection dataset;

[0093] Step 2: Construct an open-world object detection model using an improved Fast R-CNN network. The open-world object detection model uses ResNet-50 as the backbone network to extract initial multi-scale feature maps; perform weighted progressive fusion on the initial multi-scale feature maps to obtain fused multi-scale feature maps; perform cross-attention calculation on the fused multi-scale feature maps to obtain fixed-size feature maps; and calculate category classification scores, bounding box regression scores, and object detection scores on the fixed-size feature maps.

[0094] Step 3: Estimate the uncertainty of the category classification score, calculate the variance of the category classification score using Monte Carlo sampling as the confidence level of the classification result, and quantify the uncertainty based on this confidence level; determine whether the category corresponding to the category classification score belongs to a known category or an unknown category based on the uncertainty; and obtain the final category classification score by backdoor adjustment based on the uncertainty.

[0095] Step 4: Train the open-world object detection model using the open-world object detection dataset.

[0096] In this embodiment of the application, step 1, obtaining an open-world object detection dataset and preprocessing the data in the open-world object detection dataset, includes:

[0097] Step (1.1) Obtain the open-world object detection dataset. The open-world object detection dataset contains natural images with labeled object instances. The open-world object detection dataset has multiple known categories and several unknown categories.

[0098] Step (1.2) involves cropping the natural images in the open-world object detection dataset to obtain images of a preset size; and modifying the instance information in the open-world object detection dataset to fit the images of the preset size.

[0099] Step (1.3) involves marking the image of the preset size, re-dividing it into several tasks, and performing incremental learning on the tasks.

[0100] In this embodiment of the application, step 2 includes:

[0101] Step (2.1) defines the feature maps of the backbone network ResNet50 at different scales as {C1,C2,C3,C4,C5}, where C1 corresponds to the features of the initial convolutional layer and pooling layer, and C2,C3,C4,C5 correspond to the features of the subsequent layers.

[0102] In step (2.2), C1 is substituted into a convolutional layer, a batch normalization layer, and a ReLU activation function for processing to obtain C2;

[0103] Substituting C2 into a convolutional layer, a batch normalization layer, and a ReLU activation function, we obtain C3;

[0104] C3 is processed by substituting it into a convolutional layer, a batch normalization layer, and a ReLU activation function to obtain C4;

[0105] C4 is processed by substituting it into a convolutional layer, a batch normalization layer, and a ReLU activation function to obtain C5;

[0106] In step (2.3), global max pooling is applied to C1, reducing the width and height of C1 to 1 / 4 of their original values. Convolution with stride of 1 and 1x1 is applied to C2 to obtain the fused feature map P2.

[0107] Step (2.4) applies a 1x1 convolution and bilinear interpolation to C2 for upsampling, so that the spatial dimensions of C2 are aligned with those of C3, resulting in C2′; and applies a 2x2 convolution with a stride of 2 to C3, so that C3 is aligned with the feature dimensions of C2′.

[0108] Step (2.5): Calculate the weights of C2′ and C3 and perform weighted fusion to obtain the fused feature map P3;

[0109] Step (2.6): Repeat steps (2.4) and (2.5) above, and weightedly fuse the features of C3 and C4, and the features of C4 and C5. When weightedly fusing the features of C3 and C4, a convolution with a stride of 4 and a stride of 4x4 is applied to obtain the fused feature map P4. When weightedly fusing the features of C4 and C5, a convolution with a stride of 8 and a stride of 8x8 is applied to obtain the fused feature map P5. A set of fused multi-scale feature maps {P2, P3, P4, P5} is obtained.

[0110] Step (2.7) involves performing cross-attention calculation on the fused multi-scale feature maps to obtain a fixed-size feature map; then, the fixed-size feature map is scored for category classification, bounding box regression, and target identification.

[0111] In this embodiment of the application, step (2.6) includes:

[0112] For each pair of data in the feature map {(C1,C2),(C2,C3),(C3,C4),(C4,C5)}, perform the following steps:

[0113] Step (2.41) involves applying a convolutional layer, a batch normalization layer, and a ReLU activation function to a pair of feature maps to obtain the first feature and the second feature.

[0114] Step (2.42) involves concatenating the first and second features along the channel dimension to obtain the third feature map;

[0115] Step (2.43) involves applying a 1x1 convolution to the third feature map to calculate the weights of each level of feature map.

[0116] Step (2.44) uses the softmax function to normalize the weights of each feature map to obtain the fusion weights of each feature map;

[0117] Step (2.45): Based on the fusion weight of each feature map, the feature maps of different scales are weighted and fused to obtain the fused feature map;

[0118] Step (2.46) applies a convolutional layer, a batch normalization layer and a ReLU activation function to the fused feature map to obtain the final weighted fused feature map;

[0119] Step (2.47) combines all the final weighted fusion feature maps to obtain the final weighted multi-scale fusion feature map {P2,P3,P4,P5}.

[0120] In this embodiment, step 3 involves estimating the uncertainty of the category classification score, calculating the variance of the category classification score using Monte Carlo sampling as the confidence level of the classification result, and quantifying the uncertainty based on this confidence level. The uncertainty is then used to determine whether the category corresponding to the classification score belongs to a known category or an unknown category. Finally, backdoor adjustments are performed based on the uncertainty to obtain the final category classification score, including:

[0121] Step (3.1) involves fusing the feature map F from the i-th sampling. i Each pixel in the array undergoes N forward propagation iterations, where i = 1, 2, ..., N;

[0122] Step (3.2) involves processing the fused feature map F obtained from the i-th sampling. i Classify using classifier C to obtain the predicted probability distribution P for each category. i (c);

[0123] Step (3.3) calculates the mean μ(c) and variance σ of the predicted probability for each category c. 2 (c) , where uncertainty is obtained;

[0124]

[0125] Step (3.4): Based on the uncertainty, evaluate the classification score of each natural image to determine whether there is an unknown category; if the classification score of a natural image exceeds the first preset unknown category threshold, it is determined that there is an unknown category.

[0126] Step (3.5) uses the calculated uncertainty as a measure of uncertainty in the backdoor adjustment.

[0127] In this embodiment of the application, step (3) includes:

[0128] Step (3.51) involves performing N forward propagations on the category classification scores during the inference process of the open-world object detection model;

[0129] Step (3.52): For each category c, calculate the average probability value u(c) of all N Monte Carlo samplings;

[0130] Step (3.53): For each class c, calculate the variance σ of the predicted probabilities from all N Monte Carlo samplings. 2 (c);

[0131] Step (3.54), using u(c) and σ 2 (c) Evaluate the prediction uncertainty of each natural image in the open world object detection dataset for a specific category. If the classification score of the natural image exceeds the second preset unknown category threshold, it is determined that there is an unknown category.

[0132] Step (3.55) involves calculating the variance σ. 2 (c) As a confidence uncertainty measure for each natural image in the open-world object detection dataset, to dynamically adjust the classification score of each natural image.

[0133] In this embodiment of the application, step (3.5) includes:

[0134] (3.51) If the variance σ 2 (c) If the value exceeds the preset threshold τ, the corresponding natural image is determined to have high uncertainty;

[0135] (3.52) The variance value σ of each high-uncertainty natural image 2 (c) As an adjustment factor, it is added to the classification score of natural images;

[0136] (3.53) Calculate the classification score S′ of the natural image for each high-uncertainty sample:

[0137] S′=S+β·σ 2 (c),

[0138] Where S is the original classification score, and β is a preset coefficient;

[0139] (3.54) If the backdoor adjusted score S′ of each category c exceeds the adjusted third classification threshold, then the corresponding natural image is determined to belong to category c.

[0140] In this embodiment of the application, step (2.7), which involves performing cross-attention calculation on the fused multi-scale feature map to obtain a feature map of fixed size, includes:

[0141] (2.71) Receive input feature map f, the dimension of feature map f is (b,n,c), b is the batch size, n is the number of detection boxes, and c is the feature dimension;

[0142] (2.72), the feature dimension dim = 64 and the input resolution of feature map f is 1024;

[0143] (2.73) The feature map f is processed by depthwise separable convolution using the nn.Conv1d(dim,dim,3,padding=1,groups=dim) convolutional layer to obtain the processed feature map; the processed feature map is added to the original input feature map to form a residual connection;

[0144] (2.74) Normalize the input feature map, then use a linear layer to transform the feature map and apply the activation function, repeat this step twice;

[0145] (2.75) Based on the input feature map obtained in step (2.74), calculate the cross attention to obtain the weighted feature representation;

[0146] (2.75) The weighted feature representation is transformed by a linear layer, and the weighted feature representation is added to the residual connection to obtain the feature map y;

[0147] (2.76) The feature map y is processed by depthwise separable convolution using the nn.Conv1d(dim,dim,3,padding=1,groups=dim) convolutional layer;

[0148] (2.77) The input feature map output from step (2.76) is normalized and further processed by the MLP layer to generate a fixed-size output feature map.

[0149] In this embodiment of the application, step (2.75) includes:

[0150] The dimensions of the acquired input feature map x are expanded into (b, n, num_heads, head_dim), where b is the batch size, n is the number of detection boxes, num_heads is the number of attention heads, and head_dim is the dimension of the attention head.

[0151] Calculate self-attention for x:

[0152] Q=φ(xW Q ), (1),

[0153] K=φ(xW K (2),

[0154] V=φ(xW V (3),

[0155]

[0156] Attn normed =Norm(Attn)+x, (5),

[0157] output = Norm(MLP(Attn) normed ))+Attn normed (6),

[0158] Among them, W Q W K W V These are the weight parameters, Q, K, and V are variables in the calculation process, Norm represents the regularization function, MLP is a multilayer perceptron, and output is the final weighted feature representation.

[0159] In this embodiment of the application, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0160] In this application embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0161] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0162] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0163] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. An open-world target detection method based on progressive feature fusion and uncertainty estimation, characterized in that, include: The trained open-world object detection model is used to detect natural images of natural scenes, and to determine the classification results of known category features, the location information of known category features, the target score of known category features, the classification results of unknown category features, the location information of unknown category features, and the target score of unknown category features. The trained open-world object detection model includes: Step 1: Obtain the open-world object detection dataset and preprocess the data in the open-world object detection dataset; Step 2: Construct an open-world object detection model using an improved Fast R-CNN network. The open-world object detection model uses ResNet-50 as the backbone network to extract initial multi-scale feature maps; perform weighted progressive fusion on the initial multi-scale feature maps to obtain fused multi-scale feature maps; perform cross-attention calculation on the fused multi-scale feature maps to obtain fixed-size feature maps; and calculate category classification scores, bounding box regression scores, and object detection scores on the fixed-size feature maps. Step 3: Estimate the uncertainty of the category classification score, calculate the variance of the category classification score using Monte Carlo sampling as the confidence level of the classification result, and quantify the uncertainty based on this confidence level; determine whether the category corresponding to the category classification score belongs to a known category or an unknown category based on the uncertainty; perform backdoor adjustment based on the uncertainty to obtain the final category classification score; Step 4: Train the open-world object detection model using the open-world object detection dataset.

2. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 1, characterized in that, Step 1: Obtain the open-world object detection dataset and preprocess the data in the open-world object detection dataset, including: Step (1.1) Obtain the open-world object detection dataset. The open-world object detection dataset contains natural images with labeled object instances. The open-world object detection dataset has multiple known categories and several unknown categories. Step (1.2) involves cropping the natural images in the open-world object detection dataset to obtain images of a preset size; and modifying the instance information in the open-world object detection dataset to fit the images of the preset size. Step (1.3) involves marking the image of the preset size, re-dividing it into several tasks, and performing incremental learning on the tasks.

3. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 1, characterized in that, Step 2 includes: Step (2.1) defines the feature maps of the backbone network ResNet50 at different scales as {C1,C2,C3,C4,C5}, where C1 corresponds to the features of the initial convolutional layer and pooling layer, and C2,C3,C4,C5 correspond to the features of the subsequent layers. In step (2.2), C1 is substituted into a convolutional layer, a batch normalization layer, and a ReLU activation function for processing to obtain C2; Substituting C2 into a convolutional layer, a batch normalization layer, and a ReLU activation function, we obtain C3; C3 is processed by substituting it into a convolutional layer, a batch normalization layer, and a ReLU activation function to obtain C4; C4 is processed by substituting it into a convolutional layer, a batch normalization layer, and a ReLU activation function to obtain C5; In step (2.3), global max pooling is applied to C1, reducing the width and height of C1 to 1 / 4 of their original values. Convolution with stride of 1 and 1x1 is applied to C2 to obtain the fused feature map P2. Step (2.4) applies a 1x1 convolution and bilinear interpolation to C2 for upsampling, so that the spatial dimensions of C2 are aligned with those of C3, resulting in C2′; and applies a 2x2 convolution with a stride of 2 to C3, so that C3 is aligned with the feature dimensions of C2′. Step (2.5): Calculate the weights of C2′ and C3 and perform weighted fusion to obtain the fused feature map P3; Step (2.6): Repeat steps (2.4) and (2.5) above, and weightedly fuse the features of C3 and C4, and the features of C4 and C5. When weightedly fusing the features of C3 and C4, a convolution with a stride of 4 and a stride of 4x4 is applied to obtain the fused feature map P4. When weightedly fusing the features of C4 and C5, a convolution with a stride of 8 and a stride of 8x8 is applied to obtain the fused feature map P5. A set of fused multi-scale feature maps {P2, P3, P4, P5} is obtained. Step (2.7) involves performing cross-attention calculation on the fused multi-scale feature maps to obtain a fixed-size feature map; then, the fixed-size feature map is scored for category classification, bounding box regression, and target identification.

4. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 3, characterized in that, Step (2.6) includes: For each pair of data in the feature map {(C1,C2),(C2,C3),(C3,C4),(C4,C5)}, perform the following steps: Step (2.41) involves applying a convolutional layer, a batch normalization layer, and a ReLU activation function to a pair of feature maps to obtain the first feature and the second feature. Step (2.42) involves concatenating the first and second features along the channel dimension to obtain the third feature map; Step (2.43) involves applying a 1x1 convolution to the third feature map to calculate the weights of each level of feature map. Step (2.44) uses the softmax function to normalize the weights of each feature map to obtain the fusion weights of each feature map; Step (2.45): Based on the fusion weight of each feature map, the feature maps of different scales are weighted and fused to obtain the fused feature map; Step (2.46) applies a convolutional layer, a batch normalization layer and a ReLU activation function to the fused feature map to obtain the final weighted fused feature map; Step (2.47) combines all the final weighted fusion feature maps to obtain the final weighted multi-scale fusion feature map {P2,P3,P4,P5}.

5. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 1, characterized in that, In step 3, uncertainty is estimated for the category classification score. The variance of the category classification score is calculated using Monte Carlo sampling as the confidence level of the classification result. The uncertainty is quantified based on this confidence level. Based on uncertainty, determine whether the category classification score corresponds to a known category or an unknown category; The final category classification score is obtained by adjusting the backdoor based on uncertainty, including: Step (3.1) involves fusing the feature map F from the i-th sampling. i Each pixel in the array undergoes N forward propagation iterations, where i = 1, 2, ..., N; Step (3.2) involves processing the fused feature map F obtained from the i-th sampling. i Classify using classifier C to obtain the predicted probability distribution P for each category. i (c); Step (3.3) calculates the mean μ(c) and variance σ of the predicted probability for each category c. 2 (c) , where uncertainty is obtained; Step (3.4): Based on the uncertainty, evaluate the classification score of each natural image to determine whether there is an unknown category; if the classification score of a natural image exceeds the first preset unknown category threshold, it is determined that there is an unknown category. Step (3.5) uses the calculated uncertainty as a measure of uncertainty in the backdoor adjustment.

6. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 5, characterized in that, Step (3) includes: Step (3.51) involves performing N forward propagations on the category classification scores during the inference process of the open-world object detection model; Step (3.52): For each category c, calculate the average probability value u(c) of all N Monte Carlo samplings; Step (3.53): For each class c, calculate the variance σ of the predicted probabilities from all N Monte Carlo samplings. 2 (c); Step (3.54), using u(c) and σ 2 (c) Evaluate the prediction uncertainty of each natural image in the open world object detection dataset for a specific category. If the classification score of the natural image exceeds the second preset unknown category threshold, it is determined that there is an unknown category. Step (3.55) involves calculating the variance σ. 2 (c) As a confidence uncertainty measure for each natural image in the open-world object detection dataset, to dynamically adjust the classification score of each natural image.

7. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 5, characterized in that, Step (3.5) includes: (3.51) If the variance σ 2 (c) If the value exceeds the preset threshold τ, the corresponding natural image is determined to have high uncertainty; (3.52) The variance value σ of each high-uncertainty natural image 2 (c) As an adjustment factor, it is added to the classification score of natural images; (3.53) Calculate the classification score S′ of the natural image for each high-uncertainty sample: S′=S+β·σ 2 (c), Where S is the original classification score, and β is a preset coefficient; (3.54) If the backdoor adjusted score S′ of each category c exceeds the adjusted third classification threshold, then the corresponding natural image is determined to belong to category c.

8. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 3, characterized in that, Step (2.7) involves performing cross-attention calculations on the fused multi-scale feature maps to obtain a fixed-size feature map, including: (2.71) Receive input feature map f, the dimension of feature map f is (b,n,c), b is the batch size, n is the number of detection boxes, and c is the feature dimension; (2.72), the feature dimension dim = 64 and the input resolution of feature map f is 1024; (2.73) The feature map f is processed by depthwise separable convolution using the nn.Conv1d(dim,dim,3,padding=1,groups=dim) convolutional layer to obtain the processed feature map; the processed feature map is added to the original input feature map to form a residual connection; (2.74) Normalize the input feature map, then use a linear layer to transform the feature map and apply the activation function, repeat this step twice; (2.75) Based on the input feature map obtained in step (2.74), calculate the cross attention to obtain the weighted feature representation; (2.75) The weighted feature representation is transformed by a linear layer, and the weighted feature representation is added to the residual connection to obtain the feature map y; (2.76) The feature map y is processed by depthwise separable convolution using the nn.Conv1d(dim,dim,3,padding=1,groups=dim) convolutional layer; (2.77) The input feature map output from step (2.76) is normalized and further processed by the MLP layer to generate a fixed-size output feature map.

9. The open-world target detection method based on progressive feature fusion and uncertainty estimation according to claim 8, characterized in that, Step (2.75) includes: The dimensions of the acquired input feature map x are expanded into (b, n, num_heads, head_dim), where b is the batch size, n is the number of detection boxes, num_heads is the number of attention heads, and head_dim is the dimension of the attention head. Calculate self-attention for x: Q=φ(xW Q ), (1), K=φ(xW K ), (2), V=φ(xW V ), (3), Attn normed =Norm(Attn)+x, (5), output=Norm(MLP(Attn normed ))+Attn normed , (6), Among them, W Q W K W V These are the weight parameters, Q, K, and V are variables in the calculation process, Norm represents the regularization function, MLP is a multilayer perceptron, and output is the final weighted feature representation.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images