An open-world object detection method based on disentangled cascaded region generation network

Abstract

This paper proposes a method for open-world object detection based on a decoupled cascaded region profiling network. This method belongs to the field of open-world object detection. First, an RGB image is input into a ResNet50 model to extract feature maps of different scales. Multi-scale information is then fused using an FPN module. The fused feature maps are then fed into an improved decoupled cascaded region profiling network. The localization quality estimation and regression branches are decoupled, gradients are propagated independently, and candidate boxes are adjusted from coarse to fine scales through the cascaded network. Coordinate corrections and probabilities for candidate boxes that may contain unknown objects are generated. Then, through non-maximum suppression, candidate boxes that exceed a probability and fall below an overlap threshold are selected. Roi Align is performed to obtain the location of the unknown object in the fused feature map. These boxes are then aligned to a precise candidate box, and the candidate box's position is fine-tuned. Feature vectors corresponding to the candidate boxes are extracted and fed into a fully connected layer. Localization box regression and localization quality assessment are performed to obtain the final detection result. This method achieves a high recall rate.

Description

Technical Field

[0001] The present invention belongs to the field of open-world object detection, and specifically provides an open-world object detection method based on a decoupled cascade region generation network. Background Art

[0002] In recent years, with the development of deep learning, object detection technology has made great progress. However, traditional object detection techniques generally only work in closed-world scenarios, that is, they can only detect objects of known categories. For open-world object detection problems, that is, object detection in situations where the objects' categories are unknown, traditional methods perform poorly.

[0003] To address this issue, several open-world object detection methods based on deep learning have emerged in recent years to handle the situation of unknown category targets. However, existing methods are usually only applicable to a specific scenario and have low recall rates when detecting unknown objects with semantic offsets in other scenarios.

[0004] There are two main types of existing open-world object detection methods: object detection methods based on convolutional neural networks and object detection methods based on Transformer.

[0005] In open-world object detection based on convolutional neural networks, such as the ORE model, the region generation network (RGN) primarily uses a region generation network (RGN) to propose candidate bounding boxes, which are then compared and clustered to identify unknown objects. The quality of the candidate bounding boxes proposed by the RGN in this model is low, and the classification and regression tasks share parameters, resulting in a coupled state. However, their focus differs: classification focuses on the object's texture information, while localization focuses more on the object's edge information. This results in incomplete feature representation, affecting the recall rate of unknown objects and impairing detection.

[0006] Transformer-based methods primarily utilize end-to-end Transformer networks in conjunction with open-world object detection methods. For example, the OW-DETR model, however, lacks inductive bias, resulting in poor fit for small datasets. Furthermore, these models require significant computing resources and training time to achieve effective detection results, making them inefficient in engineering applications and, consequently, poorly generalizable to industry.

[0007] Therefore, there is an urgent need for an open-world object detection method to improve the recall and efficiency of open-world object detection. Summary of the Invention

[0008] To address the shortcomings of existing open-world object detection methods, this paper proposes an open-world object detection method based on a decoupled cascade region generation network. This method performs detection without classification, can detect unknown objects, decouples positioning box quality estimation and regression tasks, has stronger feature expression capabilities, and has a high recall rate.

[0009] The open-world object detection method based on the decoupled cascade region generation network includes the following steps:

[0010] Step 1: First, input the RGB image captured by the camera into the resnet50 model to extract feature maps of different scales;

[0011] First, the RGB image is preprocessed and input into the residual module of different stages of the resnet50 model; the deep feature map x is obtained after the residual module l+1 :

[0012] x l+1 =x l +F(x l ,W l )

[0013] Among them, x l Represents the shallow feature map, F(x l ,W l ) represents the residual function and outputs the feature map.

[0014] Then, the deep feature map x l+1 Feature maps are extracted through the model backbone to obtain feature maps of different scales on 4 stages.

[0015] The residual module includes 1*1 convolution, 3*3 convolution, Batch Norm normalization and ReLU activation function;

[0016] The convolution formula is shown below:

[0017]

[0018] Among them, x ij is the input feature map, w k is the convolution kernel weight, k represents the size of the convolution kernel, corresponding to k*k convolution, b represents the convolution bias term, y ij Represents the output feature map. H and W represent the height and width of the input feature map, respectively. i and j represent the row and column coordinates of the pixel in the input feature map, respectively.

[0019] Step 2: The feature maps of different scales are fused through the FPN module to output a high-resolution, semantically strong fused feature map.

[0020] The specific fusion process is:

[0021] The fpn module transforms the deep feature map x from top to bottom l+1 After upsampling, they are fused with the feature maps of different scales output from the bottom up; the fusion method is to add the corresponding positions, and the upsampling method uses the nearest neighbor interpolation method.

[0022] Step 3: Input the fused feature map into the improved decoupled cascade region generation network to generate the coordinate correction of the candidate box that may contain the unknown object and the probability that the candidate box contains the object;

[0023] The improved decoupled cascade region generation network has decoupled subtask parameter branches and uses parallel convolution to decouple branches, allowing gradients to propagate independently. By adding the latest convolution operators and normalization methods, the improved decoupled cascade region generation network can more effectively aggregate features of objects of different categories. Furthermore, by maintaining alignment between feature maps and candidate boxes in a cascade manner, more accurate candidate boxes are obtained.

[0024] The specific steps are as follows:

[0025] Step 301: The decoupled cascade region generation network generates a series of candidate boxes on the input fusion feature map according to prior knowledge;

[0026] The decoupled cascade region generation network consists of two stages. In the first stage, a coarse candidate box is generated. In the second stage, the candidate box is aligned with the region of interest through offset to generate a more accurate candidate box.

[0027] In the cascade design of the network, adaptive convolution and RELU activation functions are included, as shown in the following formula;

[0028] C(x i+1 )=RELU(AdaptiveConv(x i )),i=1,…,n

[0029] AdaptiveConv(x) represents adaptive convolution, C(x i+1 ) is the output of the cascade module. The first stage of the adaptive convolution is the hole convolution and the second stage is the deformable convolution.

[0030] First, in the first stage, dilated convolution is used to adapt to the transformation of the feature space and generate candidate boxes in the decoupled cascade region generation network;

[0031] The actual convolution kernel size K of the dilated convolution is as follows:

[0032] K=k+(k-1)(r-1)

[0033] Where k is the original convolution kernel size, r is the dilation ratio of the dilated convolution parameter, and the convolution kernel size of the dilated convolution is adjusted by the dilation ratio.

[0034] In the second stage, deformable convolution is used to set the shape offset, and the spatial position of the sample is changed according to the offset learned from different feature maps to form an irregular receptive field.

[0035]

[0036] Among them, y(p) and x(p) represent the positions of the output feature map and the input feature map respectively, and m ij Represents the shape offset corresponding to the position (i, j) in the convolution kernel, α k Represents the penalty term for the spatial position after sampling exceeding the target area;

[0037] Then, by decoupling the parameter branches of positioning quality estimation and positioning box regression, the model converges faster.

[0038] Each branch contains a decoupling module, which consists of 3*3 convolution, Group Norm normalization and ReLU activation function. In the last layer of the decoupling module, the 3*3 convolution is replaced by deformable convolution.

[0039] The decoupling module parameter branch is defined as follows:

[0040] G i (x)=RELU(GN(Conv(x))),i=1,…,n

[0041] Among them, when i=1 to n-1, Conv(x) represents 3*3 convolution, and when i=n, ​​convolution is replaced by deformable convolution.

[0042] Finally, the sub-task gradients are propagated separately by decoupling the convolution branches to obtain feature maps with different receptive fields corresponding to different scales;

[0043] The formula is as follows:

[0044] DC loc (x),DC reg (x) = L2Norm(G n (…(G2(G1(C(x))))))

[0045] DC loc (x),DC reg (x) represents the feature maps output by the decoupled cascade region generation network, L2Norm represents L2 normalization, and the feature maps of different stages are gradually iterated through the C(x) module.

[0046] Step 302: Input the candidate box output in step 301 into the total loss function, and perform regression of the candidate box based on the degree of overlap between the position and shape of the object in the candidate box and the real object.

[0047] The total loss function of the decoupled cascade region generation network is calculated as:

[0048]

[0049] in, is the loss function of the localization quality assessment branch The value of is the value of the localization box regression branch loss function in stage τ. λ is a hyperparameter used to adjust the weight of the localization box regression branch loss function in the total loss function; α τ is the weight function of each stage; θ Rpn ,θ Fpn is a parameter that decouples the cascaded region generation network and FPN.

[0050] Loss Function The definition of is as follows:

[0051]

[0052] Among them, p is the position of the candidate box, p* is the position of the real box, It is a smooth L1 loss function that judges the positional relationship between the candidate box and the real box based on the centerness indicator.

[0053] Loss Function The definition of is as follows:

[0054]

[0055] Among them, ρ is the Euclidean distance between the center points of the candidate box and the real box, c is the diagonal distance of the minimum closure area that can simultaneously contain the candidate box and the real box, and IoU is the intersection over union ratio indicator in target detection.

[0056] Step 303: For each region on the feature map that may contain an unknown object, corresponding to several regressed candidate boxes, calculate the coordinate correction amount of each regressed candidate box and the probability that it contains an object;

[0057] The probability calculation formula is:

[0058]

[0059] Among them, f loc(x) is the parameter of the positioning quality estimation branch. After the sigmoid activation function, it outputs a probability value between (0, 1). At this time, the larger the output probability value, the greater the probability of containing the detected object.

[0060] Step 4: After non-maximum suppression, select the candidate boxes that exceed the probability threshold and are lower than the overlap threshold, perform Roi Align, obtain the position of the unknown object in the fused feature map, align it to the precise candidate box, fine-tune the position of the candidate box, and extract the feature vector output corresponding to the candidate box;

[0061] First, Roi Align is used to divide the area that may contain objects into multiple small blocks. Each small block is evenly sampled to obtain the coordinates of the sampling points, and bilinear interpolation is used to calculate the pixels of the sampling points. Finally, the pixel area in the small block is averaged and the feature values ​​of all small blocks are combined to obtain the feature representation of the entire area containing the object.

[0062] The calculation method of pixels in each small block is as follows:

[0063]

[0064] Among them, f(x,y) represents the interpolated pixel, f(x1,y1), f(x1,y2), f(x2,y1), and f(x2,y2) represent the pixel values ​​of the four neighboring pixels in the original image, (x1,y1), (x1,y2), (x2,y1), and (x2,y2) represent the pixels in the upper left corner, lower left corner, upper right corner, and lower right corner, respectively, and x1≤x≤x2, y1≤y≤y2.

[0065] The pixel values ​​of the sampling points in the small block are averaged, as shown in the following formula:

[0066]

[0067] Among them, f(x ave ,y ave ) represents the feature pixel value of the small block after averaging, N represents the number of points sampled in the small block, f(x i ,y i ) represents the pixel value of the sampling point;

[0068] Then, by mapping the entire object area represented by the feature to the precise candidate box, fine-tuning the position of the candidate box and extracting the feature vector output;

[0069] Step 5: Input the feature vectors into the fully connected layer respectively, perform positioning box regression and positioning quality assessment, obtain the final detection result, and output the coordinates of the detection box and the corresponding confidence.

[0070] The advantages of the present invention are:

[0071] The present invention discloses an open-world target detection method based on a decoupled cascaded region generation network. Compared with existing methods, the present invention utilizes decoupling branches to improve the expression ability of the region generation network, improves the detection recall rate of unknown objects through cascade modules, and utilizes the inductive bias of the convolution operator to learn the shape features of unknown objects in a small data set for positioning regression, thereby improving the recall rate and efficiency of open-world target detection and having strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a schematic diagram of an open-world object detection method based on a decoupled cascaded region generation network according to the present invention;

[0073] Figure 2 A schematic diagram of a decoupled cascade region generation network in the present invention;

[0074] Figure 3 This is a flow chart of an open-world object detection method based on a decoupled cascaded region generation network according to the present invention; DETAILED DESCRIPTION

[0075] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0076] The present invention provides an open world target detection method based on a decoupled cascade region generation network, such as Figure 1 and 3 The specific steps are as follows:

[0077] Step 1: First, input the RGB image captured by the camera into the resnet50 model to extract feature maps of different scales;

[0078] The specific extraction process is:

[0079] First, the RGB image is preprocessed and then input into the residual modules of different stages of resnet50;

[0080] The preprocessing is performed using a module including convolution and pooling layers;

[0081] The deep feature map x is obtained through the residual module l+1 :

[0082] x l+1 =x l +F(x l ,W l )

[0083] Among them, x l Represents the shallow feature map, F(x l ,W l ) represents the residual function and outputs the feature map.

[0084] Then, the deep feature map x l+1 Feature maps are extracted through the model backbone to obtain feature maps of different scales on 4 stages.

[0085] The residual module includes 1*1 convolution, 3*3 convolution, Batch Norm normalization and ReLU activation function;

[0086] The convolution formula is shown below:

[0087]

[0088] where x ij is the input feature map, w k is the convolution kernel weight, k represents the size of the convolution kernel, corresponding to k*k convolution, b represents the convolution bias term, y ij represents the output feature map. H and W represent the height and width of the input feature map, respectively. i and j represent the row and column coordinates of the pixel in the input feature map, respectively. This convolution formula describes the sliding process of the convolution kernel on the input matrix and the weighted sum operation performed at each position.

[0089] Step 2: The feature maps of different scales are fused through the FPN module to combine the semantic information of the high-level feature map with the position information of the low-level feature map, and output a high-resolution, strong semantic fusion feature map;

[0090] The specific fusion process is:

[0091] The fpn module transforms the deep features x from top to bottom l+1 After upsampling, they are fused with the feature maps of different scales output from the bottom up; the fusion method is to add the corresponding positions, and the upsampling method uses the nearest neighbor interpolation method.

[0092] Step 3: Input the fused feature map into the improved decoupled cascade region generation network to generate the coordinate correction of the candidate box that may contain the unknown object and the probability that the candidate box contains the object;

[0093] like Figure 2As shown in the figure, compared with the existing region generation network, the sub-task parameter branches of the improved decoupled cascade region generation network are decoupled, and the gradients are propagated independently through parallel convolution decoupling branches; by adding the latest convolution operators and normalization methods, such as deformable convolution and group normalization, the region generation network can more effectively aggregate features of objects of different categories; and, the feature map and candidate box can be aligned in a cascade manner to obtain more accurate candidate boxes.

[0094] The specific steps are as follows:

[0095] Step 301: The decoupled cascade region generation network generates a series of candidate boxes on the input fusion feature map according to prior knowledge;

[0096] The decoupled cascade region generation network consists of two stages. In the first stage, a coarse selection of candidate boxes is generated. In the second stage, the candidate boxes are aligned with the region of interest through offset, generating more accurate candidate boxes. This process, from coarse to fine, improves the recall rate of detecting unknown objects.

[0097] In the cascade design of the network, adaptive convolution and ReLU activation functions are included, as shown in the following formula;

[0098] C(x i+1 )=RELU(AdaptiveConv(x i )),i=1,…,n

[0099] AdaptiveConv(x) represents adaptive convolution, C(x i+1 ) is the output of the cascade module. The first stage of the adaptive convolution is a dilated convolution, while the second stage is a deformable convolution. Adaptive convolution is used to adjust the candidate boxes generated at each detection stage. The receptive field size is dynamically adjusted according to the different scales of the input image to obtain more accurate candidate boxes.

[0100] First, in the first stage, dilated convolution is used to adapt to the transformation of the feature space and generate candidate boxes in the decoupled cascade region generation network;

[0101] The actual convolution kernel size K of the dilated convolution is as follows:

[0102] K=k+(k-1)(r-1)

[0103] Where k is the original convolution kernel size, r is the dilated convolution parameter dilation ratio, the calculation formula of dilated convolution is the same as the convolution formula in step 1, and the convolution kernel size is adjusted by the dilation ratio.

[0104] In the second stage, deformable convolution is used to set the shape offset, and the spatial position of the sample is changed according to the offset learned from different feature maps to form an irregular receptive field.

[0105]

[0106] Among them, y(p) and x(p) represent the positions of the output feature map and the input feature map respectively, and m ij Represents the shape offset corresponding to the position (i, j) in the convolution kernel, α k Represents the penalty term for the spatial position after sampling exceeding the target area, w k represents the convolution kernel weight, and b represents the convolution bias.

[0107] Then, by decoupling the parameter branches of different subtasks, such as the localization quality estimation branch and the localization bounding box regression branch, the subtask gradients propagate independently without interfering with each other, enabling faster model convergence. Each branch contains a decoupling module consisting of a 3x3 convolution, Group Norm normalization, and a ReLU activation function. In the last layer of the decoupling module, the 3x3 convolution is replaced with a deformable convolution, making the object detector robust to objects of different scales.

[0108] The decoupling module parameter branch is defined as follows:

[0109] G i (x)=RELU(GN(Conv(x))),i=1,…,n

[0110] Among them, when i=1 to n-1, Conv(x) represents 3*3 convolution, and when i=n, ​​convolution is replaced by deformable convolution.

[0111] Finally, the decoupled convolution branch is used to propagate the subtask gradients separately, obtaining feature maps with different receptive fields at different scales. The current decoupled cascade parameter branch replaces the coupled convolution parameter branch in the original region generation network, which can decouple the propagation gradients of different subtasks and better aggregate the features of different objects.

[0112] DC loc (x),DC reg (x) = L2Norm(G n (…(G2(G1(C(x))))))

[0113] DC loc (x),DC reg (x) represents the feature maps output by the decoupled cascade region generation network, L2Norm represents L2 normalization, and the feature maps of different stages are gradually iterated through the C(x) module.

[0114] Step 302: Input the candidate box output in step 301 into the total loss function, and perform regression of the candidate box based on the degree of overlap between the position and shape of the object in the candidate box and the real object.

[0115] The candidate boxes are input into the total loss function for correction. As the loss value becomes smaller, the candidate boxes are closer to the true value boxes of the real objects. At this time, multiple candidate boxes are generated in the area where there may be objects on the feature map to ensure coverage.

[0116] The total loss function of the decoupled cascade region generation network is calculated as:

[0117]

[0118] in, is the loss function of the localization quality assessment branch The value of is the value of the localization box regression branch loss function in stage τ. λ is a hyperparameter used to adjust the weight of the localization box regression branch loss function in the total loss function; α τ is the weight function of each stage; θ Rpn ,θ Fpn is a parameter that decouples the cascaded region generation network and FPN.

[0119] Loss Function The definition of is as follows:

[0120]

[0121] Among them, p is the position of the candidate box, p* is the position of the real box, It is a smooth L1 loss function that judges the positional relationship between the candidate box and the real box based on the centerness indicator.

[0122] Loss Function The definition of is as follows:

[0123]

[0124] Among them, ρ is the Euclidean distance between the center points of the candidate box and the real box, c is the diagonal distance of the minimum closure area that can simultaneously contain the candidate box and the real box, and IoU is the intersection over union ratio indicator in target detection.

[0125] Step 303: For each region on the feature map that may contain an unknown object, corresponding to several regressed candidate boxes, calculate the coordinate correction amount of each regressed candidate box and the probability that it contains an object;

[0126] The probability calculation formula is:

[0127]

[0128] Among them, f loc(x) is the parameter of the positioning quality estimation branch. After the sigmoid activation function, it can output a probability value between (0, 1). At this time, the larger the output probability value, the greater the probability of containing the detected object.

[0129] Step 4: After non-maximum suppression, retain high-quality candidate boxes and perform Roi Align to obtain the position of the unknown object in the fused feature map, align it to the precise candidate box, fine-tune the position of the candidate box, and extract the feature vector output corresponding to the candidate box;

[0130] Specifically, after non-maximum suppression, according to the intersection-over-union ratio and probability between candidate frames, the candidate frames with higher probability and lower overlap are retained to improve the network operation efficiency, as shown in the following formula:

[0131]

[0132] Among them, Anchor Box is the candidate box, M is the candidate box with the largest output probability value, S is the multiple candidate boxes corresponding to the same object-containing area on the feature map, IoU is the intersection-over-union method, and thr is the screening threshold for non-maximum suppression.

[0133] At this point, the area corresponding to the candidate box in the feature map is likely to be the area containing the object. ROI Align is performed on this area. Roi Align is used to divide the area that may contain the object output in step 3 into multiple small blocks. Each small block is uniformly sampled to obtain the coordinates of the sampling points, and the bilinear interpolation method is used to calculate the pixels of the sampling points. Finally, the pixel area in the small block is averaged, and the feature values ​​of all small blocks are combined to obtain the feature representation of the entire area containing the object.

[0134] The specific method for calculating pixels in each small block is as follows:

[0135]

[0136] Among them, f(x,y) represents the interpolated pixel, f(x1,y1), f(x1,y2), f(x2,y1), and f(x2,y2) represent the pixel values ​​of the four neighboring pixels in the original image, (x1,y1), (x1,y2), (x2,y1), and (x2,y2) represent the pixels in the upper left corner, lower left corner, upper right corner, and lower right corner, respectively, and x1≤x≤x2, y1≤y≤y2.

[0137] Then take the average of the pixel values ​​of the sampling points in the small block, as shown in the following formula:

[0138]

[0139] Among them, f(x ave ,yave ) represents the feature pixel value of the small block after averaging, N represents the number of points sampled in the small block, f(x i ,y i ) represents the pixel value of the sampling point;

[0140] By extracting important areas in the feature map through Roi Align, we can avoid the mismatch between the area and the candidate box caused by the loss of quantization accuracy.

[0141] After extracting regional features from the candidate frame using Roi Align, use The loss function trains two sub-task detection heads. It can obtain the location of unknown objects in the fused feature map and align them to precise candidate boxes. It then fine-tunes the location of the candidate boxes for further bounding box regression and localization quality estimation, and extracts the feature vector output corresponding to the candidate boxes.

[0142] Then, the feature vector corresponding to the candidate box is extracted and output.

[0143] Step 5: Input the feature vectors obtained in step 4 into the fully connected layer respectively to perform positioning box regression and positioning quality assessment to obtain the final detection result, and output the detection box coordinates and corresponding confidence.

[0144] Example:

[0145] The embodiment includes the following steps:

[0146] In step S1, the RGB image is first input into the backbone to extract image features and obtain a feature map. Then, the feature maps of different levels are combined through the neck module to obtain a feature map with multi-scale information.

[0147] The specific steps are: first, by inputting the RGB image into the ResNet50 model, feature maps of different step sizes are output; then, through the FPN module, the semantic information of the high-level feature map and the position information of the low-level feature map are combined to output high-resolution, strong semantic feature maps; the details are as follows:

[0148] S1.1, the resnet50 model is used to extract image feature maps. The input is first preprocessed through a module containing convolution and pooling layers, and then input into the residual module of different stages. The residual module contains a 1*1 convolution, a 3*3 convolution, Batch Norm normalization and ReLU activation function. The convolution formula is shown below:

[0149]

[0150] Where f is the input matrix, g is the convolution kernel, (x, y) is the coordinate on the input matrix, (m, n) is the coordinate on the convolution kernel, and the formula represents the sliding process of the convolution kernel on the input matrix and the weighted sum operation performed at each position.

[0151] Through the model backbone, feature maps of different step sizes on four stages are output respectively, and then input into FPN for multi-scale information fusion.

[0152] S1.2. The FPN network is divided into two parts: bottom-up and top-down. Feature extraction is a bottom-up process. Four stages of ResNet50 are used to extract four feature maps of different scales. Then, from top to bottom, the deep features are upsampled to a resolution corresponding to the bottom-down output. The output feature map is fused with the top-down features by adding corresponding positions. The upsampling method uses the nearest neighbor interpolation method, as shown in the following formula:

[0153] Assuming that a feature map of w*h pixels needs to be enlarged to W*H using the nearest neighbor interpolation method, the calculation method for each pixel (dstX, dstY) in the new feature map is:

[0154]

[0155]

[0156] The pixel value at point (srcX, srcY) is equal to the pixel value at point (dstX, dstY) multiplied by a scaling factor, and only the nearest integer pixel is taken by rounding down.

[0157] The coordinate transformation calculation formula of the nearest neighbor interpolation method is as follows:

[0158]

[0159]

[0160] Where dstX and dstY are the horizontal and vertical coordinates of a pixel in the target feature map, dstWidth and dstHeight are the width and height of the target feature map, srcWidth and srcHeight are the width and height of the original feature map, and srcX and srcY are the coordinates of the original image corresponding to the target feature map at that point (dstX, dstY).

[0161] In step S2, the extracted feature maps are fed into an improved decoupled cascaded region generation network. This network generates several candidate boxes for each region that may contain an unknown object. The coordinate correction for each candidate box is calculated, as well as the probability that it contains an object.

[0162] Specifically, first, the feature map is input into the decoupled cascade region generation network to generate several candidate boxes; then, the overlap between the candidate boxes and the real object positions and shapes in the loss function is calculated to perform candidate box regression, such as Figure 1 shown.

[0163] S2.1. In the improved decoupled cascade region generation network, it is necessary to generate a series of candidate boxes on the feature map based on prior knowledge. Adaptive convolution is used to adapt to the transformation of the feature space and solve the alignment problem of the candidate boxes and feature maps in the region generation network. In the decoupled cascade region generation network, adaptive convolution includes two stages. The first stage is the dilated convolution, and its actual convolution kernel size K is shown in the following formula:

[0164] K=k+(k-1)(r-1)

[0165] Where k is the original convolution kernel size, r is the dilated convolution parameter dilation rate, and its receptive field calculation method is consistent with the standard convolution, and the dilation rate r is used to control the size of the receptive field.

[0166] In the second stage, deformable convolution is used to set the shape offset, and the spatial position of the sample is changed according to the offset learned from different feature maps to form an irregular receptive field.

[0167]

[0168] Among them, y(p) and x(p) represent the positions of the output feature map and the input feature map respectively, and m ij Represents the shape offset corresponding to the position (i, j) in the convolution kernel, α k Represents the penalty term for the spatial position after sampling exceeding the target area, w k represents the convolution kernel weight, and b represents the convolution bias.

[0169] Then, the feature map is passed through the cascade region generation network, using a multi-stage design to generate a batch of candidate boxes in each stage and pass them to the next stage to generate more accurate candidate boxes, from coarse to fine, thereby improving the recall rate of detecting unknown objects.

[0170] In the candidate box regression stage, by decoupling the positioning quality estimation branch and the positioning box regression branch and processing the two branches in parallel, the recall rate of unknown objects can be improved. Since this open-world object detection method does not involve object classification and only detects and locates unknown objects, it can avoid the phenomenon of overfitting caused by overfitting in classification and improve generalization.

[0171] S2.2. When calculating the loss function, DIoU is selected as the loss function for positioning box regression, and the positioning box quality assessment branch is calculated using the smooth L1 loss function.

[0172] Among them, the definition of the positioning box regression loss function is as follows:

[0173]

[0174] Among them, ρ is the Euclidean distance between the center points of the candidate box and the real box, c is the diagonal distance of the minimum closure area that can simultaneously contain the candidate box and the real box, and IoU is the intersection over union ratio indicator in target detection.

[0175] The definition of the positioning box quality assessment branch loss function is as follows:

[0176]

[0177]

[0178] Among them, p is the position of the candidate box, p* is the position of the real box, smoothL1 is the smooth L1 loss function, and the positional relationship between the candidate box and the real box is judged according to the centerness indicator. t, b, l, and r are the distances from the center of the candidate box to the center of the real box target, respectively.

[0179] During the training process, the values ​​of the above two loss functions are calculated at the same time, and then the total loss function is calculated at the end by the following formula:

[0180]

[0181] in, is the loss function of the localization quality assessment branch The value of is the value of the localization box regression branch loss function in stage τ. λ is a hyperparameter used to adjust the weight of the localization box regression branch loss function in the total loss function; α τ is the weight function of each stage; θ Rpn ,θ Fpn is a parameter that decouples the cascaded region generation network and FPN.

[0182] In step S3, non-maximum suppression is performed on the high-quality candidate boxes in step S2, and then Roi Align is performed to extract feature maps that may contain objects, align the candidate boxes, fine-tune the coordinate values ​​of the candidate boxes, and extract the feature vectors corresponding to the candidate boxes.

[0183] Specifically, first, the non-maximum suppression method is used to filter multiple candidate boxes corresponding to the same area containing the object, as shown in the following formula:

[0184]

[0185] Among them, Anchor Box is the candidate box, M is the candidate box with the largest output probability value, S is the multiple candidate boxes corresponding to the same object-containing area on the feature map, IoU is the intersection-over-union method, and thr is the screening threshold for non-maximum suppression.

[0186] Roi Align is used to extract regional features from the obtained high-quality candidate boxes. Then, the real annotation boxes are matched to the candidate boxes, and the detection head is trained using the smooth L1 loss function. Finally, the candidate box coordinate correction values ​​and the corresponding confidence scores are output.

[0187] Among them, the back propagation formula of Roi Align is as follows:

[0188]

[0189] Where d(.) represents the distance between pixels, Δh and Δw represent x i and i * (r, j) is the difference between the horizontal and vertical coordinates, and x j Represents the point on the feature map before pooling, y rj Represents the jth point in the rth region after pooling, i * (r,j) represents the source of the pixel.

[0190] Then, the feature vector corresponding to the candidate box is extracted in the feature map.

[0191] In step S4, the feature vector obtained in step S3 is regressed and the positioning quality is evaluated to obtain the final detection result, and the detection box coordinates and corresponding confidence levels are output.

Claims

1. A method for open-world object detection based on a decoupled cascaded region generation network, characterized in that: The following steps are involved: Step 1: First, input the collected RGB image into the resnet50 model to extract feature maps of different scales; First, the RGB image is preprocessed and input into the residual module of different stages of the resnet50 model; the deep feature map x is obtained after the residual module l+1 : x l+1 =x l +F(x l ,W l ) Among them, x l Represents the shallow feature map, F(x l ,W l ) represents the residual function and outputs the feature map; Then, the deep feature map x l+1 Feature maps are extracted through the model backbone to obtain feature maps of different scales on four stages; Step 2: The feature maps of different scales are fused through the FPN module to output a high-resolution, semantically strong fused feature map. Step 3: Input the fused feature map into the improved decoupled cascade region generation network to generate the coordinate correction of the candidate box that may contain the unknown object and the probability that the candidate box contains the object; The specific steps are as follows: Step 301: The decoupled cascade region generation network generates a series of candidate boxes on the input fusion feature map according to prior knowledge; The decoupled cascade region generation network consists of two stages. In the first stage, a coarse candidate box is generated. In the second stage, the candidate box is aligned with the region of interest through offset to generate a more accurate candidate box. Then, by decoupling the parameter branches of positioning quality estimation and positioning box regression, the model converges faster; Each branch contains a decoupling module, which consists of a 3*3 convolution, Group Norm normalization, and ReLU activation function. In the last layer of the decoupling module, the 3*3 convolution is replaced by a deformable convolution. The decoupling module parameter branch is defined as follows: G i (x)=RELU(GN(Conv(x))),i=1,…,n Among them, when i=1 to n-1, Conv(x) represents 3*3 convolution, and when i=n, ​​convolution is replaced by deformable convolution; Finally, the sub-task gradients are propagated separately by decoupling the convolution branches to obtain feature maps with different receptive fields corresponding to different scales; The formula is as follows: DC loc (x),DC reg (x)=L2Norm(G n (…(G2(G1(C(x)))))) DC loc (x),DC reg (x) represents the feature maps output by the decoupled cascade region generation network, L2Norm represents L2 normalization, and C(x) is the cascade module. Through the C(x) module, the feature maps of different stages are gradually iterated; Step 302: Input the candidate box output in step 301 into the total loss function, and perform regression of the candidate box based on the degree of overlap between the position and shape of the object in the candidate box and the real object; The total loss function of the decoupled cascade region generation network is calculated as: in, is the loss function of the localization quality assessment branch The value of is the value of the positioning box regression branch loss function in stage τ; λ is a hyperparameter used to adjust the weight of the positioning box regression branch loss function in the total loss function; α τ is the weight function of each stage, θ Rpn ,θ Fpn is the parameter of the decoupled cascaded region generation network and fpn; Step 303: For each region on the feature map that may contain an unknown object, corresponding to several regressed candidate boxes, calculate the coordinate correction amount of each regressed candidate box and the probability that it contains an object; The probability calculation formula is: Among them, f loc (x) is the parameter of the positioning quality estimation branch. After the sigmoid activation function, it outputs a probability value between (0, 1). At this time, the larger the output probability value, the greater the probability of containing the detected object; Step 4: After non-maximum suppression, select the candidate boxes that exceed the probability threshold and are lower than the overlap threshold, execute RoiAlign, obtain the position of the unknown object in the fused feature map, align it to the precise candidate box, fine-tune the position of the candidate box, and extract the feature vector output corresponding to the candidate box; Step 5: Input the feature vectors into the fully connected layer respectively, perform positioning box regression and positioning quality assessment, obtain the final detection result, and output the coordinates of the detection box and the corresponding confidence.

2. The open-world object detection method based on a decoupled cascaded region generation network according to claim 1, characterized in that: In step 1, the residual module includes 1*1 convolution, 3*3 convolution, Batch Norm normalization and ReLU activation function; The convolution formula is shown below: where x ij is the input feature map, w k is the convolution kernel weight, k represents the size of the convolution kernel, corresponding to k*k convolution, b represents the convolution bias term, y ij Represents the output feature map; H and W represent the height and width of the input feature map, respectively; i and j represent the row and column coordinates of the pixel in the input feature map, respectively.

3. The open-world object detection method based on a decoupled cascaded region generation network according to claim 1, characterized in that: The specific process of multi-scale information fusion described in step 2 is as follows: The fpn module transforms the deep feature map x from top to bottom l+1 After upsampling, they are fused with the feature maps of different scales output from the bottom up; the fusion method is to add the corresponding positions, and the upsampling method uses the nearest neighbor interpolation method.

4. The open-world object detection method based on a decoupled cascaded region generation network according to claim 1, characterized in that: The improved decoupled cascaded region generation network described in step 3 has decoupled subtask parameter branches and uses parallel convolution to decouple branches, allowing gradients to propagate independently. By adding the latest convolution operators and normalization methods, the improved decoupled cascaded region generation network can more effectively aggregate features of objects of different categories. In addition, the feature map and candidate box are aligned through cascading to obtain more accurate candidate boxes.

5. The open-world object detection method based on a decoupled cascaded region generation network according to claim 1, characterized in that: In step 3, the cascade design of the network includes adaptive convolution and RELU activation functions, as shown in the following formula: C(x i+1 )=RELU(AdaptiveConv(x i )),i=1,…,n AdaptiveConv(x) represents adaptive convolution, C(x i+1 ) is the output of the cascade module, whose first stage is the dilated convolution and the second stage is the deformable convolution; First, in the first stage, dilated convolution is used to adapt to the transformation of the feature space and generate candidate boxes in the decoupled cascade region generation network; The actual convolution kernel size K of the dilated convolution is as follows: K=k+(k-1)(r-1) Where k is the original convolution kernel size, r is the dilation ratio of the dilated convolution parameter, and the convolution kernel size of the dilated convolution is adjusted by the dilation ratio; In the second stage, deformable convolution is used to set the shape offset. According to the offset learned from different feature maps, the spatial position of the sample is changed to form an irregular receptive field. Among them, y(p) and x(p) represent the positions of the output feature map and the input feature map respectively, and m ij Represents the shape offset corresponding to the position (i, j) in the convolution kernel, α k Represents the penalty term for the spatial position after sampling exceeding the target area.

6. The open-world object detection method based on a decoupled cascaded region generation network according to claim 1, characterized in that: In step 3, the loss function The definition of is as follows: Where p is the position of the predicted box, p* is the position of the real box, It is a smooth L1 loss function that judges the positional relationship between the candidate box and the real box based on the centerness indicator; Loss Function The definition of is as follows: Among them, ρ is the Euclidean distance between the center point of the candidate box and the real box, c is the diagonal distance of the minimum closure area that can simultaneously contain the predicted box and the real box, and IoU is the intersection over union ratio indicator in target detection.

7. The open-world object detection method based on a decoupled cascaded region generation network according to claim 1, characterized in that: The step 4 is specifically as follows: First, RoiAlign is used to divide the area that may contain objects into multiple small blocks. Each small block is uniformly sampled to obtain the coordinates of the sampling points, and bilinear interpolation is used to calculate the pixels of the sampling points. Finally, the pixel area within the small block is averaged and the feature values ​​of all small blocks are combined to obtain the feature representation of the entire area containing the object. The calculation method of pixels in each small block is as follows: Where f(x,y) represents the interpolated pixel, f(x1,y1), f(x1,y2), f(x2,y1), and f(x2,y2) represent the pixel values ​​of the four neighboring pixels in the original image, (x1,y1), (x1,y2), (x2,y1), and (x2,y2) represent the pixels at the upper left corner, lower left corner, upper right corner, and lower right corner, respectively, and x1≤x≤x2, y1≤y≤y2; The pixel values ​​of the sampling points in the small block are averaged, as shown in the following formula: Among them, f(x ave ,y ave ) represents the feature pixel value of the small block after averaging, N represents the number of points sampled in the small block, f(x i ,y i ) represents the pixel value of the sampling point; Then, by mapping the entire object area represented by the feature to the precise candidate box, the position of the candidate box is fine-tuned and the feature vector output is extracted.

Images