A target detection method based on dynamic weights and hierarchical query selection strategy

Abstract

This invention discloses a target detection method based on dynamic weights and a hierarchical query selection strategy. The method extracts multi-scale feature maps, then obtains the shape and mask information of these feature maps. Convolution, flattening, and dimensionality rearrangement operations are performed on the multi-scale feature maps, combined with positional encoding input to a Transformer encoder, resulting in an enhanced image feature sequence and its mask. The image feature sequence is optimized and preliminarily predicted using a classification head and a detection head to obtain class scores and positional information, generating a global mask. Dynamic weights are calculated for each layer of feature maps. The number of queries for each layer of feature maps is allocated according to the dynamic weights. Local queries and upper-layer projected queries are selected layer by layer from top to bottom. The selected queries are used as content queries to generate reference points and positional embedding vectors. The final target detection result is output. This method improves the accuracy of target detection by projecting the high-quality queries selected in the feature maps of each layer downwards.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and object detection technology, specifically an object detection method based on dynamic weights and hierarchical query selection strategy. Background Technology

[0002] Object detection is an important research area in artificial intelligence, and its task is to locate target objects in images and classify them. In recent years, the development of object detection technology has gone through three stages: traditional, deep learning dual-stage, and deep learning single-stage. In the traditional detector stage, object detection algorithms mainly relied on manual feature construction, but due to its limited feature representation capabilities, complex detection process, and low computational efficiency, it was difficult to cope with complex scenarios.

[0003] With the advent of deep convolutional neural networks, two-stage methods, represented by R-CNN, SPPNET, and Fast R-CNN, have become mainstream. Their core idea is to first generate candidate regions, then classify and regress bounding boxes for each region. However, this "propose first, classify later" approach leads to computational redundancy and process complexity, resulting in slow detection speeds and complex workflows. To simplify the process, single-stage detectors treat detection as a unified regression problem, directly predicting bounding boxes and class probabilities on the image, such as YOLO, SSD, and RetinaNet, significantly improving detection speed. However, both traditional two-stage methods and previous single-stage models rely on predefined anchor boxes and complex hand-designed components. To address this issue, the DETR model was proposed, which for the first time introduced the Transformer architecture and ensemble prediction into the field of object detection, constructing a truly end-to-end object detection framework that eliminates the need for anchor boxes and post-processing NMS. In early DETR-like models, the decoder query input to the Transformer decoder was a static embedding that did not capture any encoder features. This approach discarded the information generated by the encoder after image processing, resulting in slow model convergence and low accuracy. Therefore, Deformable DETR proposed a two-stage scheme that selects the top K encoder features from the last encoder layer according to the class score as priority options to enhance the decoder query.

[0004] However, selecting features solely based on category scores can lead to the over-selection of certain feature maps, resulting in fewer or no selection of other feature maps. This causes some feature information to be ignored, preventing the full potential of multi-scale features from being realized. Therefore, a method is urgently needed to better utilize multi-scale features, enabling the model to better understand image information. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and propose a target detection method based on dynamic weights and hierarchical query selection strategy. By analyzing the results and confidence scores after the initial screening query, the weights of each feature layer are calculated. Then, the number of queries at each level is dynamically allocated. Finally, the high-quality queries selected in the feature map of each layer are projected downward to improve the image information of the high-quality queries in other feature maps, thereby improving the accuracy of target detection.

[0006] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows:

[0007] A target detection method based on dynamic weights and hierarchical query selection strategy includes the following steps:

[0008] Step 1: Preprocess the input image and extract multi-scale feature maps through the backbone network to obtain the shape and mask information of each feature map layer;

[0009] Step 2: Perform convolution, flattening, and dimensional rearrangement operations on the multi-scale feature map, and input it into the Transformer encoder in combination with position encoding to obtain the enhanced image feature sequence and its mask;

[0010] Step 3: Optimize the image feature sequence and perform preliminary prediction using a classification head and a detection head to obtain category scores and location information, and generate a global mask through non-maximum suppression processing;

[0011] Step 4: Calculate the dynamic weights of each feature map layer based on the shape of the feature map, the image feature sequence, the category score, and the global mask;

[0012] Step 5: Allocate the number of queries for each feature map layer according to the dynamic weights, wherein each query layer includes two parts: local query and upper-layer projection query.

[0013] Step 6: Select local queries and upper-layer projection queries layer by layer from top to bottom. The upper-layer projection query is obtained by projecting the selected query from the previous layer onto the feature map of the current layer.

[0014] Step 7: Treat the selected query as a content query and generate reference points and location embedding vectors by combining its corresponding location information;

[0015] Step 8: Input the content query, reference point, and location embedding vector into the Transformer decoder, and output the final target detection result through the classification head and detection head.

[0016] Preferably, in step 1, the backbone network is ResNet-50, and the output multi-scale feature map includes feature maps of four different scales.

[0017] Preferably, in step 2, the position code is calculated using a sine function and a cosine function, as shown in the formula:

[0018]

[0019]

[0020] Where pos represents the pixel position, d model PE(·) represents the dimension of the location embedding and the location encoding calculation formula.

[0021] Preferably, the Transformer encoder comprises six encoder layers, each of which includes a multi-scale deformable attention mechanism, a feedforward network, a dropout layer, and a layer normalization operation.

[0022] Preferably, in step 3, the image feature sequence optimization method is as follows:

[0023] Each feature map is traversed sequentially, and a two-dimensional grid coordinate system is created, covering the pixel position of each pixel in the current layer's feature map. These coordinate axes are merged into a tensor of shape H, W, 2, representing the coordinates of each pixel, where H and W represent the height and width of the feature map. The effective width and height of each image in the current layer are calculated based on the obtained mask, and used as a scaling factor to normalize the grid coordinates to the effective region. The width and height of the target box containing the normalized grid coordinate values ​​are scaled based on the layer index. Then, the normalized coordinates and the width and height of the target box containing the grid coordinate values ​​are concatenated to obtain a tensor of shape N, H*W, 4. Every four elements form a quadruple (cx, cy, w, h), and each quadruple represents a candidate proposal init_proposal. Valid prediction proposals are selected based on whether each proposal value is between (0.01, 0.99). Finally, a logarithmic transformation is performed on the selected proposals, which maps the coordinates from the [0, 1] space to the real number space, to obtain the initial proposal result output_proposal, as shown in the following expression:

[0024]

[0025]

[0026] Where N represents the batch size, lvl represents the number of feature maps, and H and W represent the height and width of the feature maps, respectively.

[0027] Finally, based on the obtained mask, the invalid proposal values ​​in the initial proposal result are set to null, and the invalid regions in the image feature sequence are filled with 0 to obtain a new feature sequence.

[0028] Preferably, in step 4, the dynamic weight is calculated as follows:

[0029] First, extract the effective feature points and their class scores for each feature map layer; calculate the L2 norm for each effective feature point; multiply the class score by the L2 norm to obtain the importance score; average the importance scores of all effective feature points in each feature map layer to obtain the initial weights; normalize the initial weights of all layers to obtain the final weights for each feature map layer.

[0030] Preferably, in step 5, the number of queries for each feature map layer is obtained by multiplying the weight of that layer by the total number of queries; the highest layer feature map contains only local queries, while the remaining layers contain both local queries and upper-layer projection queries.

[0031] Preferably, in step 6, the generation of the upper-level projection query includes:

[0032] Get the center coordinates of the selected query in the previous level;

[0033] Calculate the scaling factor based on the aspect ratio of the feature maps of adjacent layers, and project the center coordinates onto the current layer;

[0034] Find the nearest 2×2 neighborhood feature point in the current layer feature map;

[0035] Valid feature points are selected as upper-level projection queries based on the mask and category score.

[0036] Preferably, in step 7, the reference point obtains a position embedding vector through sinusoidal position encoding and multilayer perceptron projection.

[0037] Preferably, in step 8, the Transformer decoder includes 6 decoder layers, each layer including a self-attention mechanism, a multi-scale deformable attention mechanism, a feedforward network, a dropout layer, and a layer normalization operation; the final prediction result is matched with the real annotation using the Hungarian matching algorithm.

[0038] This invention has the following characteristics and beneficial effects:

[0039] This method improves the query selection strategy by proposing a dynamic weight calculation module. Different weights are assigned based on the importance of each feature map layer. The number of candidate points selected for each feature map is obtained based on the weights. Then, feature points are selected on each feature map layer according to the category score. The selected feature points consist of two parts: one part is the feature points from the current layer, and the other part is the feature points projected from the previous layer. Projecting the feature points allows the model to input relevant information from other feature maps, helping the model to better capture the features of the target and improve its detection and recognition capabilities. This invention, by introducing dynamic weight calculation, selecting candidate points layer by layer, and projecting between adjacent layers, helps the model better understand object information in the image. Attached Figure Description

[0040] Figure 1 This is a framework diagram of a target detection method based on dynamic weights and hierarchical query selection strategy according to the present invention.

[0041] Figure 2 This is a flowchart of the steps of a target detection method based on dynamic weights and hierarchical query selection strategy according to the present invention.

[0042] Figure 3 This is a structural diagram of the query cross-layer projection in the target detection method based on dynamic weight and hierarchical query selection strategy of the present invention.

[0043] Figure 4 This is a flowchart of the dynamic weight calculation module in a target detection method based on dynamic weight and hierarchical query selection strategy according to the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0045] This invention proposes a target detection method based on dynamic weights and a hierarchical query selection strategy. The model constructed by this method is the Level-DETR image target detection model, which is an improvement on DDQ-DETR. DDQ-DETR mainly includes an image feature extraction module, a Transformer-based encoder, a NMS-based query selection model, a dense query auxiliary query, an NNS-based Transformer decoder, multiple prediction heads, and a denoising training strategy. After image feature extraction of the original image through a backbone network, it is input into the Transformer encoder for feature enhancement. Then, the NMS-based query selection model is used to obtain the query input to the NMS-based Transformer decoder. Finally, the target classification and detection results are obtained based on the classification and detection heads.

[0046] In the Level-DETR model proposed in this method, the NMS-based query selection model is improved, allowing queries to be selected based on their feature maps. By projecting the selected queries to the next layer, supplementary information is added to the selected queries, thereby achieving the effect of query enhancement.

[0047] This embodiment proposes a target detection method based on dynamic weights and a hierarchical query selection strategy, such as... Figure 1 and Figure 2 As shown, the specific steps are as follows:

[0048] Step 1: Preprocess the input image and extract multi-scale feature maps through the backbone network to obtain the shape and mask information of each feature map layer.

[0049] In this embodiment, the input image is first preprocessed by scaling different images in the same batch to the same size and filling smaller images in the same batch. Then, the ResNet-50 backbone network is used to extract features from the preprocessed image to obtain multi-scale feature maps, and the shape of each feature map layer and the mask information of the multi-scale feature maps are calculated.

[0050] The specific implementation process is as follows:

[0051] First, the image is scaled to a shape with a short side of 800 and a long side less than 1333. Different images in the same batch are scaled to the same shape. Smaller images in the batch are padded, and the image is horizontally flipped with a probability of 0.5. Then, the preprocessed image is input into the ResNet-50 backbone network. The ResNet-50 backbone network uses a residual connection mechanism to extract image features step by step through multiple convolutions and residual modules, finally resulting in a multi-scale image feature tensor.Feat And obtain the mask information.

[0052] Image Feat [res2, res3, res4, res5]

[0053] Among them, res2 to res5 represent feature maps of the original images at four different scales.

[0054] Specifically, multi-scale features were extracted through the backbone network, and the shape of each feature map is ( ), where N represents the batch size, C represents the number of channels, and H and W represent the width and height of the feature map, respectively.

[0055] Furthermore, two main operations are performed on the extracted multi-scale features:

[0056] Iterate through each feature map in turn to obtain the shape of each feature map, and store all shapes in the spatial shape variable;

[0057] Then, each feature map is concatenated along its second dimension to obtain the image feature tensor. The dimension of the image feature tensor is... .

[0058] Step 1 mainly involves obtaining three variables: the image feature tensor (used for calculations in the Transformer), the spatial shape (used to divide the feature map containing the features), and the mask (used to distinguish between valid and invalid regions).

[0059] Furthermore, the method for obtaining the shape and mask information of each layer of feature map is as follows: based on the width and height of the maximum resolution feature map obtained from the backbone network, the feature maps are downsampled sequentially, and the width and height are reduced to half of their original values, and finally the width and height of all feature maps are obtained.

[0060] Understandably, in this embodiment, three variables are obtained through step 1:

[0061] Image feature tensors are used for computation in Transformer;

[0062] Spatial shape, used to divide the feature map where the feature is located;

[0063] Mask information is used to distinguish between valid and invalid regions.

[0064] Step 2: Perform convolution, flattening, and dimensional rearrangement operations on the multi-scale feature map, and input the positional encoding into the Transformer encoder to obtain the enhanced image feature sequence and its mask.

[0065] In this embodiment, the image feature tensor of the multi-scale feature map obtained in step 1 is subjected to convolution, flattening and dimensional rearrangement operations. At the same time, the input multi-scale feature map is encoded in position. The flattened image feature sequence and the position code are input into the Transformer encoder to obtain the encoded and enhanced image feature sequence. The mask of the image feature sequence is obtained according to the mask information obtained in step 1.

[0066] The specific implementation process is as follows:

[0067] Step 2-1: Map the image feature tensor of the multi-scale feature map through a convolutional layer with a kernel size of 1×1 and a stride of 1 to reduce the channel dimension to 256 and align it with the input dimension of the Transformer. Then, flatten the width and height of the image into one dimension as the sequence length. Finally, rearrange the sequence length and feature dimension to obtain an image feature sequence containing all feature information.

[0068] Step 2-2: Perform position encoding on each scale feature in the image feature tensor of the multi-scale feature map. The position information is encoded by sine and cosine waves of different frequencies. The embedding vector of each position consists of a series of sine and cosine values. The frequencies of these values ​​increase or decrease according to a certain rule. Finally, the position code Pos is output. The position encoding calculation formula is shown below:

[0069]

[0070]

[0071] Where pos represents the pixel position, d model The dimension of the positional embedding is represented by PE(·), and PE(·) represents the positional encoding calculation formula. By introducing this positional encoding, the model can distinguish the relative positions of different elements in the feature sequence and form different representations in each dimension, which helps to improve the computational efficiency and performance of the model.

[0072] Step 2-3: Input the image feature sequence obtained in Step 2-1 and the position encoding obtained in Step 2-2 into the Transformer-based encoder for self-attention calculation to enhance the feature representation. This part consists of 6 encoder layers connected end to end. Each layer includes a multi-scale deformable attention mechanism, a feedforward network, a dropout layer, and a layer normalization operation. The encoder calculates the enhanced image feature sequence memory, and then the mask obtained in Step 1 is converted into the corresponding mask information.

[0073]

[0074] Where N represents the batch size, lvl represents the number of feature maps, H and W represent the height and width of the feature maps, respectively, and D represents the feature dimension.

[0075] Step 3: Optimize the image feature sequence and perform preliminary prediction using a classification head and a detection head to obtain category scores and location information, and generate a global mask through non-maximum suppression processing;

[0076] In this embodiment, the image feature sequence obtained in step 2 is predicted to obtain the corresponding location information and classification score. Non-maximum NMS processing is performed on all feature points. For feature points with similarity exceeding the threshold, mask information is added. Finally, global mask information for the entire image feature sequence is obtained.

[0077] The specific implementation process is as follows:

[0078] Step 3-1: This model method is a two-stage detection method. All the image feature sequences obtained in Step 2 need to be processed by the classification head and the detection head to generate preliminary proposals for target detection for each image feature sequence. Specifically, each feature map is traversed sequentially, and a two-dimensional grid coordinate system is created, covering the pixel position of each pixel in the current layer feature map. These coordinate axes are merged into a tensor of shape (H, W, 2) to represent the coordinates of each pixel, where H and W represent the height and width of the feature map. The effective width and height of each image in the current layer are calculated based on the mask obtained in steps 2-3, and this is used as a scaling factor to normalize the grid coordinates to the effective area. It should be noted that due to the inconsistent sizes of images in the same batch, there is a problem of padding smaller images. The filled area is the invalid area, and the effective area is the area that does not need to be filled. The width and height of the target box where the grid coordinates are normalized to the effective area are scaled based on the hierarchical index, reflecting the hierarchical perception characteristic that higher-level features correspond to larger targets and lower-level features correspond to smaller targets. The hierarchical index is the hierarchical number of the multi-scale features. The method of scaling the hierarchical index is: exponentiation by 2, with the index number as the exponent, and multiplying the result by 0.05 to obtain the scaled width and height. In this embodiment, the initial value of the width and height of the target box is 1. Then, the scaled grid coordinates and the width and height of the target bounding box containing the grid coordinate values ​​are concatenated to obtain a tensor of shape (N, H*W, 4). This can be understood as the scaled width and height shape being (N, H*W, 2), where N represents the batch size, H and W represent the width and height of the current feature level, the third dimension represents the width and height of the grid, and the coordinate shape is (N, H*W, 2), with the third dimension representing the center coordinates. The initial proposal is obtained by concatenating the two along the third dimension.

[0079] Each set of four elements represents a proposal's (cx, cy, w, h) coordinates, where the four elements represent the center coordinates along the x-axis, y-axis, width, and height. Each quadruple represents a candidate proposal `init_proposal`. Valid predicted proposals are selected based on whether each proposal value falls within the range (0.01, 0.99). Finally, a logarithmic transformation is performed on the selected proposals, mapping the coordinates from the [0, 1] space to the real number space, resulting in the initial proposal result `output_proposal`. This transformation makes the proposals numerically more stable and easier to optimize.

[0080]

[0081]

[0082] Where N represents the batch size, lvl represents the number of feature maps, and H and W represent the height and width of the feature maps, respectively.

[0083] Step 3-2: To prevent interference from the filling areas of smaller images in the same batch, the invalid proposal values ​​are set to null based on the mask obtained in step 2-3, and the invalid areas in the image feature sequence are filled with 0 to obtain a new feature sequence.

[0084] Step 3-3: First, input the new feature sequence into a unified prediction classification head to obtain the classification score of each pixel across all categories. Then, select the maximum score from all categories as the category score for that pixel. class Each pixel's category score is considered as the top-ranked feature and is provided to the decoder as the confidence score for content queries. The feature sequence is then input into the initial object detection head and combined with the initial proposal results obtained in step 3-2 to obtain the initial prediction box. The initial prediction box is then normalized and converted into coordinate form (x1, y1, x2, y2) to obtain the final prediction box. The position information is then predicted based on the encoder's output.

[0085] Steps 3-4: Sort all feature points in descending order according to their category scores, and then perform Non-Maximum Suppression (NMS) processing. Starting from the category score, traverse sequentially and calculate the Intersection over Union (IOU) of the target boxes between the current feature point and the remaining feature points. Add a mask to feature points that exceed the threshold of 0.8 to indicate that the feature point is discarded. Finally, obtain the global mask information.

[0086] Step 4: Calculate the dynamic weights of each feature map layer based on the shape of the feature map, the image feature sequence, the category score, and the global mask.

[0087] In this embodiment, based on the shape of the feature map obtained in step 1, the image feature sequence obtained in step 2, and the category score and mask obtained in step 3, the number of feature points retained in each feature map layer and the mask information of each feature map layer are first obtained according to the mask. Then, the corresponding feature vector is obtained from the image feature sequence through the mask information, and the L2 norm of each feature vector is calculated. Then, the importance score is calculated according to the category score and L2 norm of the feature points. Finally, the average importance score of the effective feature points of each feature map layer is used as the weight of that layer, and the weights of all feature maps are normalized.

[0088] like Figure 4 As shown, the specific implementation process is as follows:

[0089] Step 4-1: First, based on the shape of the feature map, obtain the boundary range of each feature map layer. From the image feature sequence, category score, and mask, obtain the feature sequence, category score, and mask corresponding to the feature map layer based on the boundary range.

[0090] Step 4-2: Calculate the L2 norm of the feature sequence of the feature map of this layer. Based on the effective feature sequence of the obtained feature map, calculate the L2 norm for each effective feature point, square each element, sum all the squared values, and finally take the square root of the sum.

[0091] Step 4-3: Multiply the L2 norm of the obtained feature point by the corresponding class score to obtain the importance score of the feature point. Then, calculate the average importance of the feature map of this layer as the initial weight of the layer. Finally, normalize the initial weights of all feature maps to obtain the final weights of each layer's feature map. The formula for calculating the feature map weights is as follows:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] Where N represents the batch size and lvl represents the number of feature maps. The score represents the score of valid feature points, mask represents the feature point mask, h and w represent the width and height of each feature map, D represents the feature dimension, L2 represents the Euclidean distance calculated on the feature vector, count represents the number of valid feature points in the feature map, and weight represents the feature map weight.

[0098] Step 5: Allocate the number of queries for each feature map layer according to the dynamic weights, where each query includes two parts: a local query and a projection query from the upper layer.

[0099] Specifically, such as Figure 3 As shown, in this embodiment, based on the shape of the feature map obtained in step 1 and the image feature sequence obtained in step 2, the image feature sequence is used to obtain the image feature sequence of each layer of feature maps according to the shape of the feature map. The starting index and ending index of each feature map are obtained by defining the range of each layer of feature maps obtained in step 4-1. Based on the feature map weights obtained in step 4-3 and the total number K of model configurations, the number K to be allocated to each layer is calculated. level The query for each feature map layer is divided into two parts. local and Query upper Since the highest-level feature map has no features from higher levels, this layer's feature map is generated by the query. local The lower three-layer feature map is divided into two parts, and each part contains the number of feature maps in each layer.

[0100]

[0101] Where weight represents the feature map weights, and K represents the number of queries input to the decoder.

[0102] Step 6: Select local queries and upper-level projection queries layer by layer from top to bottom. The upper-level projection query is obtained by projecting the selected query from the previous layer onto the feature map of the current layer.

[0103] Specifically, in this embodiment, after step 5, the allocation configuration of each layer of feature maps is obtained. Selection is performed from top to bottom, taking the number of features selected from each part of each layer of feature maps from the configuration. Based on the category scores generated by the classification head, a corresponding number of features are selected from the four layers of feature maps as the query. local Query in the lower three-layer feature map upper The selection process involves projecting the selected query onto the previous layer's feature map. First, the center coordinates of the selected query from the previous layer are calculated. Then, a scaling factor is calculated based on the aspect ratio of adjacent feature maps. The center coordinates of the previous layer are then projected onto the current layer. The nearest feature point is found in the current feature map, and a 2x2 neighborhood is generated. After the projection operation is complete, based on the mask information, a corresponding number of features are selected from the effective region according to the classification score generated by the classification head as the query. upper .

[0104] The specific implementation process is as follows:

[0105] Step 6-1: Based on the allocation configuration of each layer of feature maps obtained in Step 5, select all feature maps from top to bottom, and extract the number of selected features from the two parts of the feature map from the allocation configuration.

[0106] Step 6-2: Based on the category scores of the feature vectors obtained in Step 3, the mask of the feature map obtained in Step 4, the image feature sequence of each layer of feature maps obtained in Step 5, and the number of selected elements for each part of each layer of feature maps obtained in Step 6-1, first, sort the feature maps of this layer in descending order according to the mask and category scores, select the index of the first corresponding number of elements, and then extract the corresponding feature vectors from the image feature sequence to form the Query. local .

[0107] Step 6-3 In the lower three-layer feature map, perform Query projection operation. Based on the position information of the feature points obtained in step (3) and the Query selected in the previous layer, first obtain the center coordinates of the prediction box based on the position information, then multiply the center coordinates by the width and height of the current feature map to obtain the true center coordinates. Then calculate the width-to-height ratio of the two feature maps based on the shape of the feature map obtained in step (2) to obtain the scaling factors of the width and height. Then multiply the x-axis coordinate of the center coordinates by the scaling factor of the width and the y-axis coordinates by the scaling factor of the height to obtain the new center coordinates.

[0108] Step 6-4 first generates the offset of the 2×2 neighborhood. [[0,0],[0,1],[1,0],[1,1]], add the offset to the center coordinates to get the projected center coordinates. Based on the starting index of each feature map obtained in step (5), add the result of the y-axis coordinate of the projected center coordinate multiplied by the width of the current feature map, and add the x-axis coordinate of the projected center coordinate to get the final projected index.

[0109] After obtaining the complete set of indices in step 6-5, duplicate indices are removed, and invalid indices are removed using a mask. Finally, the category scores are extracted based on the indices, sorted in descending order, and the number of indices selected is determined. If the remaining number of indices is less than the number to be selected, all remaining indices are selected. Then, the corresponding feature vectors are extracted from the image feature sequence to form the query. upper Finally, the two parts are merged into the current layer's query.

[0110] Step 6-6 If the total number of items selected at the end is less than the preset K, then select from the remaining indexes according to the category score and perform the fill operation.

[0111] Step 7: Treat the selected query as a content query and generate reference points and location embedding vectors by combining its corresponding location information.

[0112] Specifically, the Query selected in step 6 is used as the content query, and reference points are generated based on the location information obtained in step 3. The reference points are then embedded by generating periodic sinusoidal positions.

[0113] The specific implementation process is as follows:

[0114] Based on the Query obtained in step 6, the content query is used as input to the Transformer decoder. The corresponding target box representation is extracted from the location information according to the index of the content query to generate a reference point, which is used as the geometric coordinate prior in the Transformer decoder. The reference point is encoded by sinusoidal position and then projected by MLP to obtain the corresponding position embedding vector.

[0115] Step 8: Input the content query, reference point, and location embedding vector into the Transformer decoder, and output the final target detection result through the classification head and detection head.

[0116] The content query obtained in step 6 and the reference point and position embedding vector obtained in step 7 are input into the Transformer decoder for calculation. After prediction by the classification head and the detection head, all target coordinate boxes and category information are obtained. Hungarian matching is then performed to obtain the final prediction result.

[0117] The specific implementation process is as follows:

[0118] The content query obtained in step 6, the reference point and location embedding vector obtained in step 7 are input into a Transformer decoder consisting of 6 layers. Each layer of the decoder includes a self-attention mechanism, a multi-scale deformable attention mechanism, a feedforward network, a dropout layer, and layer normalization. The self-attention mechanism models global relationships among all queries, enabling information exchange between different targets. The multi-scale deformable attention mechanism uses the sampling positions of reference points at different feature scales to adaptively aggregate attention across multiple feature maps, allowing the model to simultaneously focus on large and small targets. After iterative processing through the 6 layers of the decoder, each query vector gradually focuses on a specific target region, ultimately yielding a query feature representation. These feature representations are then combined with a classification head and a bounding box detection head to output the category prediction and target box coordinates for each query. Finally, the model uses the Hungarian matching algorithm to establish a one-to-one correspondence between the prediction results and the ground truth labels, thereby obtaining the final detection result and achieving end-to-end target detection.

[0119] The object detection method based on dynamic weights and hierarchical query selection strategy provided in this embodiment was implemented in a Python 3.8 (Ubuntu 18.04) and CUDA 11.3 experimental environment. The hardware environment was an RTX 4080 Super, an 11th Gen Intel(R) Core(TM) i7-11700, using the AdamW optimizer, BASE_LR of 0.0002, a batch size of 2, and a training batch size of 12. This experiment used ResNet-50 as the backbone network, and configured the decoder and encoder layers to have 6 layers, with 300 queries.

[0120] In this embodiment, 7393 images were randomly selected from the MS COCO dataset for training and 313 images were used for validation. The extracted data was guaranteed to contain 80 categories. The MS COCO dataset is a large-scale dataset that can be used for image detection and is divided into 80 target categories (pedestrians, cars, elephants, etc.).

[0121] Table 1. Comparison of the overall AP performance of the present invention and the original model in the target detection task.

[0122]

[0123] As can be seen from Table 1, compared with the original DDQ-DETR model, the method of allocating queries by hierarchy in this invention has improved all indicators except AP@75 and APm when the projection range is 2×2. When the projection range is 4×4, all indicators except APm have improved.

[0124] Experimental results show that the DETR target detection model that allocates queries according to hierarchy in this invention improves the overall detection accuracy of the model.

[0125] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A target detection method based on dynamic weight and hierarchical query selection strategy, characterized in that, The method comprises the following steps: Step 1, preprocessing the input image, and extracting multi-scale feature maps through a backbone network to obtain picture feature tensors, shapes and mask information of each layer feature map; Step 2, performing convolution, flattening and dimension rearrangement operations on the picture feature tensors of the multi-scale feature maps, combining position encoding to input a Transformer encoder to obtain enhanced image feature sequences and their masks; Step 3, optimizing the image feature sequences and performing preliminary prediction through a classification head and a detection head to obtain category scores and position information, and generating a global mask through non-maximum suppression processing; The image feature sequence optimization method is: Each feature map is traversed in turn, creating a two-dimensional grid coordinate system covering each pixel position of the current level feature map, and merging these coordinate axes into a tensor shaped H, W, 2 , representing the coordinates of each pixel, where H, W represents the height and width of the feature map. According to the obtained mask, the effective width and height of each image at the current layer are calculated, and used as scaling factors to normalize the grid coordinates to the effective area. The width and height of the target box in which the value of the grid coordinate normalized to the effective area is located are scaled based on the level index. Then, the scaled grid coordinates and the width and height of the target box in which the grid coordinate value is located are spliced to obtain a tensor shaped N, H*W, 4 , every four elements of which form a four-tuple composed of the center coordinate x-axis, y-axis, and width and height. Each four-tuple represents a candidate proposal. According to whether each proposal value is located between the preset threshold, the valid prediction proposals are screened out. Finally, a logarithmic transformation is performed on the screened proposals. The transformation maps the coordinates from the [0, 1] space to the real number space to obtain the initial proposal result. Finally, invalid proposal values in the initial proposal results are set to null values according to the obtained mask, and invalid regions in the image feature sequence are filled with 0 to obtain a new feature sequence; Step 4, calculating dynamic weights of each layer feature map based on the shapes of the feature maps, the image feature sequences, the category scores and the global mask; Step 5, distributing the number of queries of each layer feature map according to the dynamic weights, wherein each layer query includes local queries and upper layer projection queries; Step 6, selecting local queries and upper layer projection queries layer by layer from top to bottom, wherein the upper layer projection query is obtained by projecting the selected query of the previous layer to the current layer feature map; The generation of the upper layer projection query comprises: obtaining the center coordinates of the selected query of the previous layer; calculating a scaling factor according to the width-height ratio of adjacent layer feature maps and projecting the center coordinates to the current layer; finding 2*2 neighborhood feature points closest to the projection coordinates in the current layer feature map; selecting valid feature points as the upper layer projection query according to the mask and the category score; Step 7, taking the selected query as a content query, combining its corresponding position information to generate a reference point and a position embedding vector; Step 8, inputting the content query, the reference point and the position embedding vector into a Transformer decoder to output the final target detection result through a classification head and a detection head.

2. The method of claim 1, wherein, In step 1, the backbone network is ResNet-50, and the output multi-scale feature maps include four feature maps of different scales.

3. The method of claim 1, wherein, In step 2, the position encoding is calculated by using a sine function and a cosine function.

4. The method of claim 1, wherein, The Transformer encoder comprises 6 encoder layers, and each encoder layer comprises a multi-scale deformable attention mechanism, a feedforward network, a dropout layer and a layer normalization operation.

5. The method of claim 1, wherein, In step 4, the calculation method of the dynamic weights is as follows: first, extracting valid feature points and their category scores of each layer feature map; calculating the L2 norm of each valid feature point; multiplying the category score and the L2 norm to obtain an importance score; averaging the importance scores of all valid feature points of each layer feature map to obtain an initial weight; normalizing the initial weights of all layers to obtain the final weight of each layer feature map.

6. The method of claim 1, wherein, In step 5, the number of queries of each layer feature map is obtained by multiplying the weight of the layer and the total number of queries; the highest layer feature map only contains local queries, and the remaining layers contain local queries and upper layer projection queries.

7. The method of claim 1, wherein, In step 7, the reference points are projected to position embedding vectors by sinusoidal position encoding and multi-layer perceptron.

8. The method of claim 1, wherein, In step 8, the Transformer decoder contains 6 decoder layers, each of which includes a self-attention mechanism, a multi-scale deformable attention mechanism, a feedforward network, a dropout layer, and a layer normalization operation; the final prediction result is matched with the true label by the Hungarian matching algorithm.

Images