Target detection method and electronic device
By using a target detection method based on a deformable attention model and leveraging textual prompts containing scale information for small target detection, the problems of low resolution and strict localization accuracy requirements are solved, thereby improving the accuracy of small target detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SAIC GENERAL MOTORS
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-12
AI Technical Summary
Small object detection in computer vision faces challenges such as low resolution, scarce visual details, susceptibility to noise interference, and stringent requirements for positioning accuracy, making detection difficult.
A target detection method based on a deformable attention model is adopted, which uses text cues containing scale information to detect images. Feature extraction and localization are performed through a backbone network, a text encoding module, an encoder layer, a decoder layer, and a detection head. Training and optimization are carried out by combining text embedding branch and center point localization branch.
It improves the accuracy and semantic understanding of small target detection, achieves better feature extraction and localization, and enhances the accuracy of small target description.
Smart Images
Figure CN121661334B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology in computer vision, and in particular to a target detection method, electronic device, storage medium, and computer program product. Background Technology
[0002] Small target detection is a technique in computer vision used to identify and locate targets that occupy a small percentage of the image's pixels. It is widely used in the transportation sector. Small targets are common in automotive scenarios, such as traffic signs, traffic lights, and distant pedestrians and vehicles in autonomous driving environments. The accuracy of small target detection directly impacts traffic safety.
[0003] The core challenges of small object detection lie in two interconnected key aspects. First, small objects, due to their low resolution and sparse visual details, are more susceptible to image noise, further obscuring key features. Coupled with often insufficient contextual information, this makes it difficult for models to extract sufficiently robust and discriminative feature representations. Second, the accuracy requirements for small object localization are extremely stringent; a one-pixel offset in the bounding box of a small object has a far greater relative error impact than that of a large object. These two combined challenges make small object detection a major hurdle in the field of computer vision. Summary of the Invention
[0004] Therefore, it is necessary to provide a target detection method, electronic device, storage medium, and computer program product.
[0005] This invention provides a target detection method, comprising:
[0006] Obtain images and dataset text prompts for inference, wherein the dataset text prompts include multiple text prompts containing scale information;
[0007] The image used for inference and the text prompts in the dataset are input into the object detection model to obtain the predicted object type and predicted object location output by the object detection model. The object detection model is built based on a deformable attention model and uses text prompts containing scale information to perform object detection on the image.
[0008] Furthermore, the target detection model includes: a backbone network, a text encoding module, an encoder layer based on a deformable attention model, a decoder layer, and a detection head, wherein:
[0009] The backbone network is used to extract image features at multiple scales from the image and input the image features at multiple scales into the encoder layer.
[0010] The text encoding module is used to obtain dataset text prompts, encode the dataset text prompts to obtain text embeddings, and add the text embeddings to the original target query to obtain a target query containing scale information. The original target query includes multiple query features.
[0011] The encoder layer is used to encode image features at multiple scales to obtain image feature codes at multiple scales;
[0012] The decoder layer is used to interact with the target query by encoding image features at multiple scales output by the encoder layer using an attention mechanism, thereby obtaining multiple decoder layer output features.
[0013] The detection head is used to output the predicted target category and the predicted target location based on the output features of the decoder layer.
[0014] Furthermore, the scale information includes multiple scale types. Encoding the dataset text prompts to obtain text embeddings, and adding the text embeddings to the original target query to obtain a target query containing scale information, includes:
[0015] The text prompts in the dataset are encoded to obtain text embeddings;
[0016] The first learnable weight matrices of multiple scale types are concatenated to obtain a concatenated weight matrix. The concatenated weight matrix is then multiplied with the text embedding to obtain the predicted text embedding.
[0017] The predicted text embedding is added to the original target query to obtain a target query containing scale information.
[0018] Furthermore, the attention mechanism is used to interact with the multi-scale image feature encoding output from the encoder layer and each target query containing scale information to obtain multiple decoder layer output features, including:
[0019] For each target query that includes scale information, perform the following operation:
[0020] The target query is multiplied by the second learnable weight matrix of each scale information to obtain a set of normalized offsets for each scale. The number of normalized offsets is different for each scale, and the smaller the scale, the larger the number of normalized offsets.
[0021] Based on the reference position corresponding to the target query, the normalized offset of the scale is superimposed on each scale to obtain the sampling point of each scale;
[0022] From the image feature encoding at each scale, obtain the image feature encoding for each sampling point;
[0023] The image feature encoding of each scale sampling point is multiplied by the third learnable weight matrix of each scale to obtain the attention value vector;
[0024] Attention fusion is performed based on the target query and attention value vector to obtain the decoder layer output features.
[0025] Furthermore, the object detection model also includes a text embedding branch, which is used to output the predicted text embedding.
[0026] Furthermore, the target detection model also includes a center point localization branch, which is used to output a predicted Gaussian heatmap.
[0027] Furthermore, it also includes:
[0028] Acquire training data, which includes images for training and target labels for training. The target labels include the target type, target location, and text prompts containing scale information for each target.
[0029] The training data is input into the object detection model for multiple iterations of training. In each iteration, the following steps are performed:
[0030] Obtain the predicted target type, predicted target location, predicted text embedding, and predicted Gaussian heatmap output by the target detection model;
[0031] Use the target type of the target tag as the target type truth value, and the target position of the target tag as the position truth value;
[0032] The text embedding obtained by encoding the text prompt of the target label is used as the text embedding truth value;
[0033] Based on the target location of the target label, the Gaussian heatmap corresponding to the image used for training is determined as the Gaussian heatmap ground value;
[0034] Calculate the type loss value based on the predicted target type and the true value of the target type;
[0035] Calculate the position loss value based on the predicted target position and the true target position;
[0036] The text loss value is calculated based on the predicted text embedding and the ground truth text embedding.
[0037] Calculate the Gaussian heatmap loss value based on the predicted Gaussian heatmap and the true value of the Gaussian heatmap;
[0038] The total loss value is calculated based on the type loss value, the location loss value, the text loss value, and the Gaussian heatmap loss value;
[0039] The target detection model is adjusted based on the total loss value, and the next iteration of training is performed.
[0040] This invention provides an electronic device, comprising:
[0041] At least one processor; and,
[0042] A memory communicatively connected to at least one of the processors; wherein,
[0043] The memory stores instructions that are executed by at least one of the processors to enable the at least one processor to perform the target detection method as described above.
[0044] The present invention provides a storage medium that stores computer instructions, which, when executed by a computer, are used to perform all the steps of the target detection method as described above.
[0045] The present invention provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the target detection method as described above.
[0046] This invention constructs a target detection model that uses text prompts containing scale information to guide target detection in images. By utilizing text prompts with scale information to guide target detection, the accuracy of small target descriptions is enhanced, and the model's semantic understanding of small targets is improved. This invention can better achieve small target feature extraction and localization, thereby improving the accuracy of small target detection. Attached Figure Description
[0047] Figure 1 This is a flowchart illustrating the process of a target detection method according to an embodiment of the present invention;
[0048] Figure 2 This is a flowchart illustrating the process of a target detection method according to another embodiment of the present invention;
[0049] Figure 3 A schematic diagram of the target detection model according to the preferred embodiment of the present invention;
[0050] Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to the present invention. Detailed Implementation
[0051] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. Identical components are indicated by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.
[0052] like Figure 1 The diagram shown is a flowchart of a target detection method according to an embodiment of the present invention, including:
[0053] Step S101: Obtain the image and dataset text prompts for reasoning, wherein the dataset text prompts include multiple text prompts containing scale information;
[0054] Step S102: Input the image used for inference and the text prompt data into the object detection model to obtain the predicted object type and predicted object location output by the object detection model. The object detection model is constructed based on a deformable attention model, and the object detection model uses text prompts containing scale information to perform object detection on the image.
[0055] Specifically, this invention can be applied to electronic devices with processing capabilities, such as computers, or vehicle controllers. The vehicle controller can be an Electronic Control Unit (ECU).
[0056] First, step S101 is executed to obtain the image and dataset text prompts for inference. The dataset text prompts include multiple text prompts containing scale information.
[0057] The images used for reasoning can be captured by a camera device, such as a vehicle's camera.
[0058] The dataset text hints consist of multiple text hints, each containing scale information.
[0059] The scale information is categorized into three classes—large, medium, and small—based on the definition of target size in the COCO dataset. This scale information is then used to process the annotation information. Based on the scale information in the annotations, a text prompt is designed as "a photo of [size] [class]". The dataset text prompt is composed of all labels from the given target detection dataset, processed to include corresponding text prompts containing scale information. Here, [size] represents the scale information, including the three categories of large, medium, and small, while [class] represents the target type.
[0060] Then, step S102 is executed, in which the image used for inference and the text prompts of the dataset are input into the object detection model to obtain the predicted object type and predicted object location output by the object detection model. The object detection model is built based on a deformable attention model, and the object detection model uses text prompts containing scale information to perform object detection on the image.
[0061] Specifically, the object detection model is built upon a deformable attention model. Deformable attention models include, but are not limited to, the Deformable DETR (Detection Transform) model and the DINO (DETR with Improved deNoising anchOr boxes) model. The object detection model comprises: a text encoding module, a backbone network, an encoder layer based on the deformable attention model, a decoder layer, and a detection head.
[0062] The image and dataset text prompts used for inference are input into the object detection model. Since the dataset text prompts include multiple text prompts containing scale information, the object detection model can use the text prompts containing scale information to perform object detection on the image and output the predicted object type and predicted object location.
[0063] The target location is used to identify the bounding box of the predicted target. The target location uses... Method of representation, The x-coordinate of the center point of the bounding box. The Y-coordinate of the center point of the bounding box. The width of the bounding box. This represents the height of the bounding box.
[0064] This invention constructs a target detection model that uses text prompts containing scale information to guide target detection in images. By utilizing text prompts with scale information to guide target detection, the accuracy of small target descriptions is enhanced, and the model's semantic understanding of small targets is improved. This invention can better achieve small target feature extraction and localization, thereby improving the accuracy of small target detection.
[0065] like Figure 2 The diagram shown is a flowchart of a target detection method according to another embodiment of the present invention, including:
[0066] Step S201: Obtain the image and dataset text prompts for reasoning, wherein the dataset text prompts include multiple text prompts containing scale information;
[0067] Step S202: Input the image used for inference and the text prompts of the dataset into the object detection model to obtain the predicted object type and predicted object location output by the object detection model. The object detection model is constructed based on a deformable attention model, and the object detection model uses text prompts containing scale information to perform object detection on the image. The object detection model includes: a backbone network, a text encoding module, an encoder layer based on a deformable attention model, a decoder layer, a detection head, a text embedding branch, and a center point localization branch.
[0068] The backbone network is used to extract image features at multiple scales from the image and input the image features at multiple scales into the encoder layer.
[0069] The text encoding module is used to obtain dataset text prompts, encode the dataset text prompts to obtain text embeddings, and add the text embeddings to the original target query to obtain a target query containing scale information. The original target query includes multiple query features.
[0070] The encoder layer is used to encode image features at multiple scales to obtain image feature codes at multiple scales;
[0071] The decoder layer is used to interact with the target query by encoding image features at multiple scales output by the encoder layer using an attention mechanism, thereby obtaining multiple decoder layer output features.
[0072] The detection head is used to output the predicted target category and the predicted target location based on the output features of the decoder layer;
[0073] The text embedding branch is used to output the predicted text embedding;
[0074] The center point positioning branch is used to output the predicted Gaussian heatmap.
[0075] Specifically, step S201 is first executed to obtain an image and dataset text prompts for inference, wherein the dataset text prompts include multiple text prompts containing scale information.
[0076] The images used for reasoning can be captured by a camera device, such as a vehicle's camera. The text prompts to be reasoned contain scale information.
[0077] The scale information is categorized into three classes—large, medium, and small—based on the definition of target size in the COCO dataset. This scale information is then used to process the annotation information. Based on the scale information in the annotations, a text prompt is designed as "a photo of [size] [class]". The dataset text prompt is composed of all labels from the given target detection dataset, processed to include corresponding text prompts containing scale information. Here, [size] represents the scale information, including the three categories of large, medium, and small, while [class] represents the target type.
[0078] Then, step S202 is executed, in which the image used for inference and the text prompts of the dataset are input into the object detection model to obtain the predicted object type and predicted object location output by the object detection model. The object detection model is constructed based on a deformable attention model, and the object detection model uses text prompts containing scale information to perform object detection on the image. The object detection model includes: a backbone network, a text encoding module, an encoder layer based on a deformable attention model, a decoder layer, a detection head, a text embedding branch, and a center point localization branch.
[0079] The backbone network is used to extract image features at multiple scales from the image and input the image features at multiple scales into the encoder layer.
[0080] The text encoding module is used to obtain dataset text prompts, encode the dataset text prompts to obtain text embeddings, and add the text embeddings to the original target query to obtain a target query containing scale information. The original target query includes multiple query features.
[0081] The encoder layer is used to encode image features at multiple scales to obtain image feature codes at multiple scales;
[0082] The decoder layer is used to interact with the target query by encoding image features at multiple scales output by the encoder layer using an attention mechanism, thereby obtaining multiple decoder layer output features.
[0083] The detection head is used to output the predicted target category and the predicted target location based on the output features of the decoder layer;
[0084] The text embedding branch is used to output the predicted text embedding;
[0085] The center point positioning branch is used to output the predicted Gaussian heatmap.
[0086] like Figure 3 The diagram shown illustrates the target detection model of the preferred embodiment of the present invention, including: a backbone network 301, a text encoding module 302, an encoder layer 303 based on a deformable attention model, a decoder layer 304, a detection head 305, a text embedding branch 306, and a center point localization branch 307. The processing flow of the target detection model is as follows:
[0087] Step 1: The image is processed by a backbone network 301 to extract multi-scale feature maps. The features in each scale feature map are the image features at that scale. These multi-scale image features are then input into the encoder layer 303. The backbone network 301 is preferably ResNet-50. Simultaneously, to incorporate positional information into the image features, positional embedding and encoding are added to the image features input into the encoder layer 303. For example, a grayscale gradient image can be constructed and directly added to the image features.
[0088] Step 2: The dataset text prompts consist of all the processed labels from the given object detection dataset. The specific steps are as follows:
[0089] Based on the definition of target size in the COCO dataset, scale information is categorized into three types: large, medium, and small. The annotation information in the target labels is processed, and text prompts are designed based on the scale information in the annotations, such as "a photo of [size][class]", "a photo of a large rider", "a photo of a medium car", "a photo of a small traffic light", etc. Multiple text prompts are then combined into a single dataset text prompt.
[0090] Step 3: Encode the text prompts from the dataset in the previous step using a pre-trained model, such as the text encoder of a Contrastive Language–Image Pre-training (CLIP) model, to obtain predicted text embeddings. These predicted text embeddings are used to construct the target query.
[0091] In one embodiment, the scale information includes multiple scale types, and the process of encoding the dataset text prompts to obtain text embeddings, and adding the text embeddings to the original target query to obtain a target query containing scale information, includes:
[0092] The text prompts in the dataset are encoded to obtain text embeddings;
[0093] The first learnable weight matrices of multiple scale types are concatenated to obtain a concatenated weight matrix. The concatenated weight matrix is then multiplied with the text embedding to obtain the predicted text embedding.
[0094] The predicted text embedding is added to the original target query to obtain the target query containing scale information.
[0095] Specifically:
[0096] (1) By generating text embeddings for the dataset text prompts through a text encoder such as the CLIP model, text embeddings for targets of different scales under different categories can be obtained;
[0097] (2) Concatenate the first learnable weight matrices of small, medium and large scales, multiply the concatenated weight matrix with the text embedding to obtain the predicted text embedding. The first learnable weight matrix is initialized using Xavier, and its dimension is [number of query features, number of target categories];
[0098] (3) The original target query is a set of learnable embedding vectors, each of which is a query feature. The dimension of the original target query is the same as that of the predicted text embedding (which contains information at different scales). The predicted text embedding and the original target query are directly added together to obtain the final target query.
[0099] Encoder layer 303 employs a deformable attention model encoder layer, such as a Deformable DETR model encoder layer or a DINO encoder layer. The encoder layer includes multiple encoders to receive image features at multiple scales.
[0100] Each encoder contains a cascaded multi-scale deformable attention module and a fully connected feedforward network. In addition, two residual connections can be added outside the multi-scale deformable attention module and the fully connected feedforward network, and normalization operations can be set after them.
[0101] Among them, the multi-scale deformable attention module uses multi-scale deformable attention to encode image features at multiple scales to obtain image feature codes for each scale.
[0102] The encoder layer can be implemented using, for example, a deformable DETR encoder layer. The backbone network 301 uses feature maps across L levels, for example, L=4. These feature maps from different stages of the backbone network are processed separately through subsequent convolutional operations. Each level of feature map has a different resolution; all features of each level's feature map represent the image features at that level's resolution. These multi-scale features are then input into the encoder layer 303 to obtain the image feature encoding for each scale of the feature image.
[0103] Step 4: Decoder layer 304 is an improved decoder layer based on the deformable attention model, such as the Deformable DETR model or the DINO decoder layer. A scale-cued attention mechanism is used to interact image features from feature maps of different scales with the target query containing scale information, i.e., attention fusion (cross attention), to obtain multiple decoder layer output features to enhance the features of small targets.
[0104] In one embodiment, the attention mechanism is used to interact with the multi-scale image feature encoding output from the encoder layer and each target query containing scale information to obtain multiple decoder layer output features, including:
[0105] For each target query that includes scale information, perform the following operation:
[0106] The target query is multiplied by the second learnable weight matrix of each scale information to obtain a set of normalized offsets for each scale. The number of normalized offsets is different for each scale, and the smaller the scale, the larger the number of normalized offsets.
[0107] Based on the reference position corresponding to the target query, the normalized offset of the scale is superimposed on each scale to obtain the sampling point of each scale;
[0108] From the image feature encoding at each scale, obtain the image feature encoding for each sampling point;
[0109] The image feature encoding of each scale sampling point is multiplied by the third learnable weight matrix of each scale to obtain the attention value vector;
[0110] Attention fusion is performed based on the target query and attention value vector to obtain the decoder layer output features.
[0111] Specifically, the decoder layer includes one or more cascaded deformable decoders. Each deformable decoder comprises a multi-head self-attention module, a multi-scale deformable cross-attention module, and a fully connected feedforward network, all connected in sequence. The multi-head self-attention module and the fully connected feedforward network both utilize existing Deformable DETR model multi-head self-attention modules and fully connected feedforward networks. The multi-scale deformable cross-attention module is also connected to the decoder layer, and each decoder's multi-scale deformable cross-attention module performs the following operations:
[0112] The target query is multiplied by the second learnable weight matrix of information at each scale to predict a set of normalized offsets for each scale. k = 1, 2, 3, ..., K, where K is the number of offsets. The second learnable weight matrix is initialized using Xavier, and its dimension is [feature dimension, number of feature layers]. [Number of sampling points] The number of normalized offsets varies for each scale, and the smaller the scale, the greater the number of normalized offsets (i.e., the number of sampling points);
[0113] At reference location A normalized offset is superimposed on top of this, and the initial reference position is obtained through linear layer prediction using the existing Deformable DETR method, based on the target query. For reference position X-axis and Y-axis coordinates, reference position Superimposed offset Then, the offset position is obtained. The offset position at each scale is the sampling point at that scale;
[0114] Since the offset is obtained through prediction, the prediction result is unlikely to be an integer. Bilinear interpolation is used to determine the image features corresponding to each sampling point. For the l-th scale, the image feature code of the sampling point corresponds to the image feature code of the location of that sampling point at that scale. For example... , where p is the sampling point in the key region of the l-th scale, and the image feature encoding comes from the encoder;
[0115] The image feature encoding of each sampling point at each scale is multiplied by the third learnable weight matrix at each scale to obtain the value V of the attention mechanism. The third learnable weight matrix is initialized using Xavier and has the dimension [feature dimension, feature dimension].
[0116] Substituting the target query Q and the value V into the deformable attention formula yields the scale-hint attention fusion value, Attention. The deformable attention formula can be the same as the Deformable DETR formula, for example: Attention = softmax(QWa)V, where Attention is the attention fusion value, Wa is the third learnable weight matrix, and the dimension is [feature dimension, number of feature layers]. Number of sampling points].
[0117] Finally, the attention fusion values are processed through operations such as FFN and normalization to obtain the final decoder output features. Specifically, the target query Q of the first decoder is the target query containing scale information obtained in step three; the target queries Q of the remaining decoders are the decoder output features of the previous decoder; and the decoder output features of the last decoder are used as the decoder layer output features.
[0118] In this embodiment, decoder layer 304 is an improvement upon the decoder layer of, for example, the Deformable DETR model. The difference lies in the fact that in the Deformable DETR model decoder, the number of offsets for each scale is fixed, for example, 4, thus the number of sampling points per scale is fixed. However, in this embodiment, the number of offsets varies for different scales; the smaller the scale, the larger the number of offsets, resulting in a larger number of sampling points and more accurate sampling.
[0119] Step 5: Using the existing deformable attention model detection head 305, such as the Deformable DETR model or the DINO model detection head, the predicted target category and predicted target location are output based on the output features of the decoder layer. At the same time, a text embedding branch 306 is added to the detection head to output the predicted text embedding from step 3, avoiding information interference caused by the mixing of features from image and text modalities. This branch participates in the training phase.
[0120] Step six involves introducing a center point localization branch 307, which outputs a predicted Gaussian heatmap. This corrected Gaussian heatmap is used to refine the target center point localization, improving the accuracy of small target localization. This branch participates in the training phase. Specifically, the predicted Gaussian heatmap is a fixed-size learnable matrix (heatmap). This learnable matrix is used to compare with the ground truth Gaussian heatmap during the training phase to improve center point localization.
[0121] In one embodiment, it further includes:
[0122] Acquire training data, which includes images for training and target labels for training. The target labels include the target type, target location, and text prompts containing scale information for each target.
[0123] The training data is input into the object detection model for multiple iterations of training. In each iteration, the following steps are performed:
[0124] Obtain the predicted target type, predicted target location, predicted text embedding, and predicted Gaussian heatmap output by the target detection model;
[0125] Use the target type of the target tag as the target type truth value, and the target position of the target tag as the position truth value;
[0126] The text embedding obtained by encoding the text prompt of the target label is used as the text embedding truth value;
[0127] Based on the target location of the target label, the Gaussian heatmap corresponding to the image used for training is determined as the Gaussian heatmap ground value;
[0128] Calculate the type loss value based on the predicted target type and the true value of the target type;
[0129] Calculate the position loss value based on the predicted target position and the true target position;
[0130] The text loss value is calculated based on the predicted text embedding and the ground truth text embedding.
[0131] Calculate the Gaussian heatmap loss value based on the predicted Gaussian heatmap and the true value of the Gaussian heatmap;
[0132] The total loss value is calculated based on the type loss value, the location loss value, the text loss value, and the Gaussian heatmap loss value;
[0133] The target detection model is adjusted based on the total loss value, and the next iteration of training is performed.
[0134] Specifically, the classic Hungarian algorithm is used for matching. Besides the target category and bounding box coordinates, the Hungarian matching strategy and loss function also include the calculation of center point probability and text embedding. The Hungarian matching algorithm is a prediction strategy for DETR-type model ensembles. Because a new branch has been added, information from both branches needs to be incorporated into the Hungarian matching. The Hungarian algorithm is as follows:
[0135] Where Lcost is the matching cost function, which can be implemented using a loss function of the corresponding type, y i The i-th truth value is one of the following: target type truth value, location truth value, text embedding truth value, or Gaussian heatmap truth value. Let be the i-th predicted value in the σ solution, i.e., one of the following in the σ solution: predicted target type, predicted target location, predicted text embedding, and predicted Gaussian heatmap. N is the number of predicted values. This is the optimal solution.
[0136] Based on the predicted target type and the true value of the target type, the type loss value is calculated using an existing type loss function. Based on the predicted target location and the true target location, the location loss value is calculated using an existing location loss function.
[0137] During model training, the ground truth text embeddings can be obtained using text cues containing scale information from the input target label, through steps two and three. Specifically, the scale-information text cues are input into the CLIP model to obtain the output ground truth text embeddings, which are then compared with the predicted text embeddings output by the text embedding branch. The text loss value is calculated using, for example, an L1 loss function.
[0138] Based on the predicted Gaussian heatmap and the true Gaussian heatmap, the Gaussian heatmap loss value is calculated using the existing focus loss function.
[0139] The specific steps are as follows:
[0140] (1) Based on the target location in the target label used for training, determine the bounding box of each target in the image used for training, and scale each bounding box. The scaled bounding box ensures that the scale difference between the large and small targets is as small as possible. Its mathematical expression is:
[0141]
[0142] Where H0 is the height of the image used for training, and W0 is the width of the image. These are the bounding box coordinates before scaling. The x-axis coordinates of the center point of the bounding box before scaling. The Y-axis coordinate of the center point of the bounding box before scaling. This is the width of the bounding box before scaling. H represents the height of the bounding box before scaling. hm To predict the height of the Gaussian heatmap, W hm To predict the width of the Gaussian heatmap, the scaled bounding box coordinates are: x is the X-axis coordinate of the center point of the scaled bounding box, y is the Y-axis coordinate of the center point of the scaled bounding box, w is the width of the scaled bounding box, and h is the height of the scaled bounding box.
[0143] (2) Based on the scaled bounding box, design the Gaussian radius to ensure the effective presentation of the center point positioning.
[0144] (3) Design the Gaussian heatmap of each bounding box using the Gaussian radius, and use the Gaussian heatmap to represent the probability of the center point. The design of the Gaussian radius and the Gaussian heatmap can be implemented using Cornernet.
[0145] (4) Based on the position of the Gaussian heatmap of each bounding box in the image, map the Gaussian heatmap of each bounding box to a map of the same size as the predicted Gaussian heatmap to obtain the ground truth Gaussian heatmap of the entire image.
[0146] (5) Based on the predicted Gaussian heatmap and the true value of the Gaussian heatmap, the Gaussian heatmap loss value is calculated using the existing focal loss function formula.
[0147] This embodiment improves upon existing deformable attention models, such as the Deformable DETR model or the DINO model. Scale information is added to the input text prompts, and the decoder layer of existing deformable attention models is improved by setting different numbers of sampling points based on different scales, with a larger number of sampling points for smaller scales. Simultaneously, during the training phase, a text embedding branch and a center point localization branch are added to improve training accuracy.
[0148] This embodiment employs a scale-cueing Transformer and a center point localization branch to improve the accuracy of small object detection. The scale-cueing Transformer enhances small object features by introducing textual prompts containing scale information, enriching the target query information with the help of a pre-trained model, and thus guiding the calculation of scale-cueing attention. Simultaneously, a text embedding branch is added to avoid information interference caused by features from both image and text modalities. Finally, this embodiment uses a center point localization branch to correct and refine the localization of small objects. The target center is refined into a Gaussian heatmap based on the target bounding box, and a target size scaling method is used to correct the localization differences of targets of different sizes, alleviating the sensitivity to small object localization. This embodiment can better achieve small object feature extraction and localization, thereby improving the accuracy of small object detection.
[0149] Table 1 compares the detection metrics of the algorithm of this invention with other algorithms on the BDD100K dataset. All compared algorithms employ multi-scale input and can be categorized into DETR-based and non-DETR-based models. The standard Average Precision (AP) metric is used to comprehensively evaluate the performance of the object detection models, along with AP. 50 AP 75 Hierarchical metrics such as APS, APM, and APL are used to effectively identify performance differences of the model in different scenarios.
[0150] Table 1
[0151]
[0152] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0153] like Figure 4 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising:
[0154] At least one processor 401; and,
[0155] A memory 402 is communicatively connected to at least one of the processors 401; wherein,
[0156] The memory 402 stores instructions that can be executed by at least one of the processors to enable the at least one of the processors to perform the target detection method as described above.
[0157] Figure 4 Take a processor 401 as an example.
[0158] The electronic device may also include an input device 403 and a display device 404.
[0159] The processor 401, memory 402, input device 403 and display device 404 can be connected by a bus or other means. The figure shows an example of connection by bus.
[0160] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the target detection method in the embodiments of this application, for example, Figure 1 , Figure 2 The method flow is shown. The processor 401 executes various functional applications and data processing by running non-volatile software programs, instructions, and modules stored in the memory 402, thereby realizing the target detection method in the above embodiments.
[0161] The memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the target detection method, etc. Furthermore, the memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may optionally include memory remotely located relative to the processor 401, and these remote memories can be connected to the apparatus performing the target detection method via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0162] The input device 403 can receive user clicks and generate signal inputs related to user settings and function control of the target detection method. The display device 404 may include a display screen or other display device.
[0163] The one or more modules are stored in the memory 402, and when run by the one or more processors 401, they execute the target detection method in any of the above method embodiments.
[0164] This invention constructs a target detection model that uses text prompts containing scale information to guide target detection in images. By utilizing text prompts with scale information to guide target detection, the accuracy of small target descriptions is enhanced, and the model's semantic understanding of small targets is improved. This invention can better achieve small target feature extraction and localization, thereby improving the accuracy of small target detection.
[0165] One embodiment of the present invention provides a storage medium that stores computer instructions, which, when executed by a computer, are used to perform all the steps of the target detection method as described above.
[0166] In the context of this disclosure, a storage medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The storage medium can be a machine-readable signal medium or a machine-readable storage medium. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage device.
[0167] One embodiment of the present invention provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the target detection method as described above.
[0168] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A target detection method, characterized in that, include: Obtain images and dataset text prompts for inference, wherein the dataset text prompts include multiple text prompts containing scale information; The image used for inference and the text prompts in the dataset are input into the object detection model to obtain the predicted object type and predicted object location output by the object detection model. The object detection model is built based on a deformable attention model and uses text prompts containing scale information to perform object detection on the image. The object detection model includes: a backbone network, a text encoding module, an encoder layer based on a deformable attention model, a decoder layer, and a detection head, wherein: The backbone network is used to extract image features at multiple scales from the image and input the image features at multiple scales into the encoder layer. The text encoding module is used to obtain dataset text prompts, encode the dataset text prompts to obtain text embeddings, and add the text embeddings to the original target query to obtain a target query containing scale information. The original target query includes multiple query features. The encoder layer is used to encode image features at multiple scales to obtain image feature codes at multiple scales; The decoder layer is used to interact with the target query by encoding image features at multiple scales output by the encoder layer using an attention mechanism, thereby obtaining multiple decoder layer output features. The detection head is used to output the predicted target category and the predicted target location based on the output features of the decoder layer.
2. The target detection method according to claim 1, characterized in that, The scale information includes multiple scale types. Encoding the dataset text prompts to obtain text embeddings, and adding the text embeddings to the original target query to obtain a target query containing scale information, includes: The text prompts in the dataset are encoded to obtain text embeddings; The first learnable weight matrices of multiple scale types are concatenated to obtain a concatenated weight matrix. The concatenated weight matrix is then multiplied with the text embedding to obtain the predicted text embedding. The predicted text embedding is added to the original target query to obtain a target query containing scale information.
3. The target detection method according to claim 1, characterized in that, The attention mechanism is used to encode image features at multiple scales from the encoder layer and interact with each target query containing scale information to obtain multiple decoder layer output features, including: For each target query that includes scale information, perform the following operation: The target query is multiplied by the second learnable weight matrix of each scale information to obtain a set of normalized offsets for each scale. The number of normalized offsets is different for each scale, and the smaller the scale, the larger the number of normalized offsets. Based on the reference position corresponding to the target query, the normalized offset of the scale is superimposed on each scale to obtain the sampling point of each scale; From the image feature encoding at each scale, obtain the image feature encoding for each sampling point; The image feature encoding of each scale sampling point is multiplied by the third learnable weight matrix of each scale to obtain the attention value vector; Attention fusion is performed based on the target query and attention value vector to obtain the decoder layer output features.
4. The target detection method according to claim 1, characterized in that, The object detection model also includes a text embedding branch, which is used to output the predicted text embedding.
5. The target detection method according to claim 4, characterized in that, The target detection model also includes a center point localization branch, which is used to output a predicted Gaussian heatmap.
6. The target detection method according to claim 5, characterized in that, Also includes: Acquire training data, which includes images for training and target labels for training. The target labels include the target type, target location, and text prompts containing scale information for each target. The training data is input into the object detection model for multiple iterations of training. In each iteration, the following steps are performed: Obtain the predicted target type, predicted target location, predicted text embedding, and predicted Gaussian heatmap output by the target detection model; Use the target type of the target tag as the target type truth value, and the target position of the target tag as the position truth value; The text embedding obtained by encoding the text prompt of the target label is used as the text embedding truth value; Based on the target location of the target label, the Gaussian heatmap corresponding to the image used for training is determined as the Gaussian heatmap ground value; Calculate the type loss value based on the predicted target type and the true value of the target type; Calculate the location loss value based on the predicted target location and the true location value; The text loss value is calculated based on the predicted text embedding and the ground truth text embedding. Calculate the Gaussian heatmap loss value based on the predicted Gaussian heatmap and the true value of the Gaussian heatmap; The total loss value is calculated based on the type loss value, the location loss value, the text loss value, and the Gaussian heatmap loss value; The target detection model is adjusted based on the total loss value, and the next iteration of training is performed.
7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that are executed by at least one of the processors to enable the at least one of the processors to perform the target detection method as described in any one of claims 1 to 6.
8. A storage medium, characterized in that, The storage medium stores computer instructions, which, when executed by a computer, are used to perform all the steps of the target detection method as described in any one of claims 1 to 6.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the target detection method as described in any one of claims 1 to 6.