Target object detection method and device, electronic equipment and storage medium

Abstract

This application provides a method, apparatus, electronic device, and storage medium for detecting target objects. The method includes: obtaining the original features of a pre-labeled reference object in the original image based on the feature map of the input original image; interacting the original features with the feature map to obtain a feature response map; generating a prediction density map and a prediction angle map based on the feature response map; and labeling the target object corresponding to the reference object based on the confidence that each first pixel in the prediction density map is the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features. The target object matching the reference object can be detected through the pre-labeled reference object, avoiding the problem of inaccurate target object detection caused by insufficient training sample diversity and poor generalization ability, and improving the accuracy and generalization of target object detection.

Description

Technical Field

[0001] This application belongs to the field of industrial target recognition technology, specifically relating to a target object detection method, device, electronic equipment, and storage medium. Background Technology

[0002] In related technologies, the task of rotating target detection and counting in industrial overhead shooting scenarios relies on a large number of training samples. Due to factors such as large changes in target rotation angle and complex target distribution, traditional methods rely on a large number of training samples to cover various situations. When the samples are insufficient, the model has difficulty accurately capturing the target, which leads to an increase in the rate of missed detection or false detection. It is easy to have problems such as missed detection, false detection or insufficient generalization ability, resulting in inaccurate target detection. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and storage medium for detecting target objects, which can solve the problem of inaccurate target detection.

[0004] In a first aspect, embodiments of this application provide a method for detecting a target object. The method includes: obtaining original features of a pre-marked reference object in the original image based on a feature map of an input original image; performing information interaction between the original features and the feature map to obtain a feature response map; generating a prediction density map and a prediction angle map based on the feature response map; and marking the target object corresponding to the reference object based on the confidence that each first pixel in the prediction density map is the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features.

[0005] Secondly, embodiments of this application provide a target object detection device, the device comprising: an acquisition module, configured to acquire original features of a pre-marked reference object in the original image based on a feature map of an input original image; and an interaction module, configured to interact the original features with the feature map to obtain a feature response map; The generation module is used to generate a prediction density map and a prediction angle map based on the feature response map; The detection module is used to mark the target object corresponding to the reference object based on the confidence that each first pixel in the predicted density map is the center point of the target object, the predicted angle represented by each second pixel in the predicted angle map, and the width and height features in the original features.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] In this embodiment, the original features of a pre-labeled reference object in the original image are obtained based on the feature map of the input original image; the original features are interacted with the feature map to obtain a feature response map; a prediction density map and a prediction angle map are generated based on the feature response map; and the target object corresponding to the reference object is labeled based on the confidence that each first pixel in the prediction density map is the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features. The target object matching the reference object can be detected through the pre-labeled reference object, avoiding the problem of inaccurate target object detection caused by insufficient training sample diversity and poor generalization ability, and improving the accuracy and generalization of target object detection. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic flowchart of a target object detection method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a target object detection device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0013] The following description, in conjunction with the accompanying drawings, details the target object detection method, apparatus, electronic device, and storage medium provided in this application through specific embodiments and application scenarios.

[0014] Figure 1 This application illustrates an embodiment of a method for detecting a target object, which can be executed by an electronic device including a model for detecting the target object. In other words, the method can be executed by software or hardware installed in the electronic device, and includes the following steps: Step S101: Based on the feature map of the input original image, obtain the original features of the reference object pre-labeled in the original image.

[0015] Object detection technology has been widely applied in practical scenarios such as industrial automation, intelligent manufacturing, and video surveillance. Especially in industrial overhead shots, target objects (such as parts, components, and products) are typically neatly arranged and uniform in size, but their orientation is not fixed, exhibiting obvious rotational characteristics. Therefore, rotated object detection has become an important technology for solving such problems. Rotated object detection refers to detecting directional target objects in an image and representing the target's position and orientation with a rotated bounding box containing angle information. This method is suitable for scenarios where targets have arbitrary orientations, such as detecting parts in overhead shots.

[0016] The target object detection method provided in this application first requires acquiring an original image of an industrial overhead scene including the object to be detected. Then, based on the feature map of the input original image, the original features of a pre-marked reference object in the original image are obtained.

[0017] In the application embodiment, before obtaining the original features of the reference object pre-labeled in the original image based on the feature map of the input original image, the method further includes: inputting the original image into a lightweight neural network backbone structure for feature extraction to obtain a feature map.

[0018] Specifically, in this embodiment, the original image is first input into a lightweight neural network backbone structure for feature extraction to obtain a feature map. This application uses EfficientViT as the backbone network, which balances computational efficiency and semantic expressive power, making it suitable for deployment in edge devices or industrial equipment as the basic input for subsequent processing.

[0019] The embodiments of this application use a lightweight backbone network (such as EfficientViT). The entire inference process only requires one forward propagation and one round of feature matching, which can run smoothly on edge devices and has good deployability and real-time performance.

[0020] In this embodiment, reference objects are pre-marked in the original image using cue boxes. These cue boxes can be rectangular boxes, and detected target objects can also be marked using rectangular boxes. These cue boxes can be manually specified or automatically generated to represent representative target regions, guiding the model to learn features of specific categories or types. In this embodiment, the cue boxes serve as input to guide the model in extracting features, representing reference objects of interest in the detection task.

[0021] In this embodiment of the application, the original features of the reference object refer to the semantic representation extracted through the prompt box area, which is used to describe the general features of the reference object.

[0022] In one embodiment, based on the feature map of the input original image, the original features of a reference object pre-labeled in the original image are obtained, including: obtaining the first position rotation feature and width and height feature of the reference object from the feature map; performing average pooling on the feature map to obtain the background feature of the entire image; and combining the first position rotation feature, width and height feature and the background feature of the entire image into the original features.

[0023] In this embodiment of the application, the original features of the reference object include the following three parts: First position rotation feature: Also known as rotational region of interest (ROI) feature, this feature extracts the target position of the reference object from the feature map output by the backbone network using a rotation-aware region cropping algorithm. This first position rotation feature reflects the local texture and surrounding information of the reference object.

[0024] Width and height features (including width and height): Considering the importance of the scale and shape of the reference object to semantics, the embodiments of this application can map the width and height features of the reference object (tooltip) into an independent high-dimensional vector to represent the height-to-width ratio and geometric information of the target. In order to prevent ambiguity, the embodiments of this application can place the higher of the width and height features first and the lower of the width and height features second.

[0025] Global background features: By performing average pooling on the semantic features of the entire feature map, a global image representation is obtained, which serves as a background feature reference and provides context awareness.

[0026] The first position rotation feature, width and height feature, and global background feature are fused to form a three-vector sequence, which represents the expression of the reference object in terms of content, structure and environment, and has good generalization and recognition guidance capabilities.

[0027] Step S102: Interact with the original features and the feature map to obtain the feature response map.

[0028] In this embodiment, the original features of the reference object are used as key values ​​in the attention mechanism to interactively encode the semantic features (feature map) of the entire original image to obtain a feature response map, thereby completing the identification and response enhancement of the region of interest.

[0029] In one embodiment, information interaction between the original features and the feature map is performed to obtain a feature response map, including: using the original features as keys and values, and using the feature map as a query to update the features, generating an initial response map, wherein each position of the initial response map includes the relevance or difference with the target object; performing self-attention enhancement on the initial response map to enhance the relevance or difference between each position of the initial response map and the target object, thereby obtaining the feature response map.

[0030] In this embodiment, a two-level attention mechanism is mainly applied. The first level of attention mechanism is prototype-guided attention: the reference object serves as a memory bank, guiding the response enhancement of the region in the feature map that matches it, thereby enabling the search for regions similar to the reference object in the entire image. That is, the feature map is used as the query, and the original features of the reference object are used as the key and value. This operation can be performed multiple times to enhance the perception capability.

[0031] The specific formula is shown below:

[0032] in, Indicates a query. Represents a predefined set of real numbers. H represents the expansion of the feature map features, where H represents the height of the original feature and W represents the width of the original feature.

[0033] Generate keys and values:

[0034] in and Indicates the preset weight shape C C and P are the three-vector sequence shapes (N, C) of the original features. The resulting Q, K (keys), and V (values) are all of shape (N, C). N equals M+2, where M represents the number of reference objects and C represents the feature dimension in deep learning. Here, C can be equal to 768.

[0035] Attention weight calculation:

[0036] in Indicates attention weights. , This represents the features at positions i and j in the feature map. This represents the scaling factor.

[0037] In this embodiment of the application, attention weights are preset. Perform feature updates:

[0038] This represents the initial response map, from which the processing results of the first-level attention mechanism are obtained.

[0039] The second-level attention mechanism is self-attention feature enhancement. After being guided by the first-level attention mechanism, the response region of the feature map is self-modeled again to further enhance the consistency and difference of related regions. That is, the original response map after guidance is subjected to multiple self-attention operations to increase the global perception of the model. The final output is called the feature response map, which is an image feature representation that integrates prototype guidance and self-attention enhancement, and is used for subsequent detection.

[0040] Specifically, the original response map obtained from the first-level attention mechanism is used as input. By modeling the correlation between the features at each location in the original response map, the features of target regions with similar semantics or spatial relationships are mutually enhanced, while the response of background region features that are unrelated to the target is weakened.

[0041] This process gradually introduces global contextual information by performing multiple self-attention operations on the guided original response map, thereby enhancing the model's overall perception of the target region and improving the consistency of features within the target region as well as the difference between the target and the background.

[0042] The output obtained after self-attention feature enhancement is defined as a feature response map. This feature response map integrates prototype guidance information and the global feature representation enhanced by self-attention, and is used as input for subsequent target detection to improve detection accuracy and robustness.

[0043] Let the original response graph of the first-level attention mechanism be... :

[0044] A linear mapping is performed on the original response graph to obtain the query. , :

[0045] Where d represents the number of calculations, and d-1 can be 0, in which case... .

[0046] Calculate attention weights:

[0047] This represents the correlation weight between spatial locations, s is the scaling factor of the feature dimension, which is equal to the number of channels C, i.e., 768, and Softmax is the Softmax function.

[0048] Finally, self-attention feature updates are performed:

[0049] Final output feature response map d represents the number of times the second-level attention mechanism is processed, and in this application, d can be 2.

[0050] In this embodiment, the purpose of the two-level attention mechanism is to allow the original features of the reference object and the feature map of the original image to interact, making regions similar to the reference object in the feature map more prominent and enhancing the response to obtain a feature response map. The prototype-guided attention part mainly uses the feature map as a query (Q) and the original features as keys (K) and values ​​(V). It can be inferred that the features at each position are compared with the original features to see if they match the original features of the reference object. The resulting feature may contain information about whether each position in the feature map contains the reference object. Self-attention feature enhancement is performed again on the enhanced original response map after prototype-guided attention. This can be understood as the interaction between the responses at each location in the original feature map and the responses at other locations. For example, if there is a large object, many locations in the original response map may indicate the presence of this object, but only a few are actually the real object. Therefore, the original response maps need to communicate with each other to determine which are the real objects and which are background noise, so as to obtain a more definite feature response map. The final feature response map is a tensor feature (B, C, H, W), which contains information such as whether there is a real object corresponding to a reference object at each location, the angle and shape of the real object, etc.

[0051] Step S103: Generate a prediction density map and a prediction angle map based on the feature response map.

[0052] In this embodiment, two output branches, density estimation and angle regression, are introduced based on the feature response map. The training process for these two output branches is as follows: First, a large multi-object dataset is constructed, including desktop photography, natural photography, and program-generated data. This dataset is trained together with the publicly available dataset FSC147. During training, the program randomly selects an image with one or more bounding boxes of different classes, each class having a varying number of instances (for example, an image with many birds and boats). Then, the program randomly selects one class (e.g., birds) from all the candidate bird bounding boxes and uses it as cue boxes. Additionally, a single-channel image with all zeros is generated, its shape and... Figure 1 To obtain the target heatmap Tdensity, a heatmap is added to this image based on the position, width, height, and orientation of all bird frames. The heatmap's value range is 0 to 1. The specific method is as follows: Target bounding box parameter definition: For a selected category (such as "bird"), let the bounding box parameter corresponding to the i-th target instance be:

[0053] in,( () represents the center coordinates of the target box. Indicates the width of the target box. Indicates the height of the target bounding box. This indicates the rotation angle of the target bounding box relative to the horizontal direction.

[0054] Coordinate rotation transformation: For any pixel (x, y) in the heatmap, first transform it to a local coordinate system with the center of the target bounding box as the origin: ( ) This rotation transformation operation is used to align the Gaussian distribution direction with the target box orientation.

[0055] Rotated Gaussian kernel response function: The standard deviation of the Gaussian kernel is defined based on the width and height of the target bounding box.

[0056] Among them, α and β are proportionality coefficients used to control the diffusion range of the heat map.

[0057] The thermal response of the i-th target box at position (x, y) is:

[0058] The response value It lies in the interval (0, 1).

[0059] When multiple target instances of the same category exist in an image, the Gaussian responses generated from all target boxes are fused to obtain the final heatmap M(x, y): M(x, y) = max

[0060] Then, the loss function is calculated using the heatmap Pdensity generated by the network. For the angle loss, the sinx and cosx values ​​of the center point of each box in the angle prediction map Pangle are taken and the actual angle of the box is calculated. Finally, gradient descent is used to reduce the loss and update the model parameters.

[0061] In one embodiment, generating a prediction density map and a prediction angle map based on the feature response map includes: processing the feature response map by bilinear interpolation upsampling and convolution to generate the prediction density map and the prediction angle map, wherein the density value of each first pixel in the prediction density map is used to represent the confidence that the first pixel is the center point of the target object, and each third pixel in the prediction angle map includes the corresponding angular direction.

[0062] In the embodiments of this application, density estimation and angle regression, both branches are composed of three layers of bilinear interpolation upsampling and convolution, which will map the feature response map back to the original image size, and the number of channels is 2.

[0063] The inference stage for generating the predicted density map and the predicted angle map includes: first, inputting the feature response map and the original features of the reference object, and saving the width and height features in the original features (if there are multiple prompt boxes, take the average width and height, and assume that the difference in width and height between each item is very small).

[0064] Then, the inference mode begins, inputting the feature response map and the original features of the reference object, ultimately obtaining the predicted density map P. density And the predicted angle diagram P angle .

[0065] Density estimation branch: Generates a predicted density map, where the value of each pixel represents the confidence that the point is the target center. Regression is performed by learning the rotated Gaussian distribution of the target center point. The rotated Gaussian distribution is normalized to 0-1 to ensure that the peak value of the response map aligns with the target center. During prediction, the density map prediction is treated as a vector; its normalized cosine value with the x-axis is calculated and divided by two to obtain the predicted value of 0-1. This value, along with the generated Gaussian density map, is used to calculate the loss.

[0066] Angle Regression Branch: Sine and cosine representations are used to predict the target's orientation to avoid discontinuities caused by angle wrapping. Each pixel predicts the potential target's angular direction, outputting two components of the angle vector. Inference uses inverse trigonometric functions to recover the actual angle. This application designs an angle correction strategy: when a target's aspect ratio is detected to be greater than 1, the target's width and height are automatically swapped, and the target bounding box is rotated 90 degrees to correct the angle, making the detection results more stable and free from angle ambiguity.

[0067] Step S104: Based on the confidence that each first pixel in the predicted density map is the center point of the target object, the predicted angle represented by each second pixel in the predicted angle map, and the width and height features in the original features, mark the target object corresponding to the reference object.

[0068] In the application embodiment, the target object corresponding to the reference object can be marked based on the confidence that each first pixel in the prediction density map is the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features.

[0069] In one embodiment, the target object corresponding to the reference object is marked based on the confidence level of each first pixel in the prediction density map as the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features. This includes: determining the center point of the target object based on the confidence level of the first pixel in the prediction density map; determining the prediction angle of the center point from the prediction angle map; and marking the target object based on the center point, the prediction angle of the center point, and the width and height features.

[0070] The specific formula is shown below: Predicted density map: P density ; Predicted angle diagram: P angle .

[0071] The predicted density map is obtained by rotating Gaussian kernel regression and has the property that local peaks correspond to instance centers.

[0072] Predicted density map center point extraction: A pixel (x, y) is defined as a candidate point for center point if and only if it satisfies:

[0073] in, This represents a local neighborhood window centered at (x, y).

[0074] The candidate set B of the center points is obtained:

[0075] Center point distance definition: For any two candidate center points ( )and( ), defining Euclidean distance :

[0076] Suppression rule: If satisfied Then, retain the center points with higher confidence levels (density values):

[0077] The final set of center points of the target object:

[0078] The output prediction angle is: P angle

[0079] The first channel represents the sine component of the angle, the second channel represents the cosine component of the angle, and B represents the batch size.

[0080] The predicted angle at position (x, y) is:

[0081] in, Represents the predicted sinθ. This represents the predicted cosθ.

[0082] Angle normalization: To ensure trigonometric function constraints, angle classification is normalized. ,

[0083] Calculation of the angle at the center point: For the nth target center point obtained from the final screening ( The corresponding prediction angle is:

[0084] in, It is a two-parameter arctangent function. .

[0085] Finally, by combining the obtained center point, the width and height features from the original features, and the predicted angle of the center point, the target object can be marked by selecting a bounding box. By predicting the probability of each type of object using a density map, and combining this with a medical perspective, target detection and counting are integrated, greatly simplifying the process and improving efficiency.

[0086] The target object detection method provided in this application involves extracting features from the original image to obtain a feature map; obtaining the original features of a pre-labeled reference object in the original image based on the feature map; interacting the original features with the feature map to obtain a feature response map; generating a prediction density map and a prediction angle map based on the feature response map; and determining the target object matching the reference object from the density map based on the confidence level of each pixel in the prediction density map as the center point of the target object, the prediction angle represented by each pixel in the prediction angle map, and the width and height features in the original features. This method detects the target object matching the reference object using the pre-labeled reference object, avoiding the problem of inaccurate target object detection due to insufficient training sample diversity and poor generalization ability, thus improving the accuracy and generalization of target object detection. This application does not rely on global category labels and only requires a single cue target to complete the detection, greatly reducing the dependence on training samples and making it suitable for small sample or weakly supervised scenarios.

[0087] In one embodiment, after marking the target object corresponding to the reference object, the method further includes: obtaining a first position rotation feature of the reference object; obtaining a second position rotation feature of the target object; calculating the similarity between the first position rotation feature and the second position rotation feature; and determining that there is a correspondence between the reference object and the target object if the similarity is greater than a similarity threshold.

[0088] In this application embodiment, to further improve the reliability of the detection results, a bounding box filtering mechanism based on feature similarity is designed. The core idea is that the reference object should not only be highlighted, but also maintain consistency with the original features of the reference object in the semantic feature space.

[0089] The implementation involves mapping semantic features from the feature response map to the contrast space using a lightweight network, constructing positive and negative sample pairs to enhance the detection of target objects. This mechanism effectively eliminates false detections with high background responses or structurally similar but semantically different results, enhancing the interpretability and practicality of the detection.

[0090] Positive sample pairs: Features extracted from all manually labeled prompt boxes are considered to belong to the same category of target regions, and the target should have the smallest distance between them in the feature space.

[0091] Negative sample pairs: Features from areas in the image not covered by any tooltip, which are different from positive samples. The goal is to maximize the feature distance between them.

[0092] By employing a loss function based on similarity distance, such as information-theoretic contrastive loss (InfoNCE), the entire feature space is structurally modulated, enabling the system to quickly locate and identify unseen target instances.

[0093] In this way, after marking the target object corresponding to the reference object, the first position rotation feature of the reference object can be obtained, the second position rotation feature of the target object can be obtained, and then the similarity between the first position rotation feature and the second position rotation feature can be calculated. If the similarity is greater than the similarity threshold, it is determined that there is a correspondence between the reference object and the target object.

[0094] The overall loss function in this embodiment consists of the following three parts: Density regression loss: The mean squared error between the predicted density map and the true rotated Gaussian map, ensuring accurate center point prediction. To encourage the model to predict the center region, a higher weight is given to the center region, and the loss is divided by the sum of the generated Gaussian density maps to balance the weights of samples of different sizes and numbers. The loss function formula is as follows:

[0095] in:

[0096] Pi is the value of the density map generated from the dataset at position i, and Pi is the predicted value of the density map at position i. The weighting function... Different weights are selected based on whether the target value is 0. In this application, fn_weight=1 and fp_weight=0.1. It is a tiny constant used to prevent division by zero.

[0097] Angular consistency loss: Based on the error between the predicted direction vector and the true angle vector, directional symmetry is considered to ensure invariance within a 180-degree rotation. Specifically, after obtaining sin(x) and cos(x), these are treated as a direction vector. The negative of the cross product of the predicted direction vector and the true angle is calculated. This value yields the maximum loss when the difference from the true angle is 90 degrees, and the minimum loss when the difference is 0 degrees or 180 degrees, thus reducing the fitting difficulty. In addition, this loss component assigns different weights to objects with different aspect ratios. Specifically, objects with high aspect ratios have higher angle weights, while objects with low aspect ratios have lower angle weights, thereby enhancing the subjective perception of prediction accuracy.

[0098]

[0099] in and These are the predicted sin(x) and cos(x), while and Let sin(x) and cos(x) be the values ​​of the target, and the weight w be calculated using the following formula:

[0100] Contrastive learning loss: A similarity loss is constructed using positive and negative sample pairs. The cosine similarity between positive samples and between positive and negative sample pairs is calculated, and the information noise contrastive estimation loss (InfoNCE Loss) is calculated. The formula is as follows:

[0101] The density regression loss and angle consistency loss are both weighted at 1, while the contrastive learning loss is weighted at 0.1.

[0102] in It is the similarity between positive samples. It represents the similarity between positive and negative sample pairs, where T is the temperature coefficient and N is the set of candidate negative sample indices.

[0103] The contrastive learning approach in this application introduces semantic intervals into the feature space and significantly suppresses false detections by using prototype alignment constraints, thereby improving detection accuracy and interpretability.

[0104] The following example illustrates the reasoning process and practical application flow for target detection in this application: In actual use, the input includes the original image and the prompt box. The overall process is as follows: 1. Preprocessing: The original image is filled with equal width and height and then scaled to 512x512 pixels, and the sample box is transformed synchronously.

[0105] 2. Original Feature Extraction: The original image is subjected to feature extraction to obtain a feature map. The feature map and bounding box information are then used to extract the original features using the extraction module. These features are saved to the system. Additionally, the ROI features of the bounding boxes in the similarity module are also saved.

[0106] 3. Forward reasoning: For each original image, the system calculates the image features and uses the feature sequence of the saved original features to obtain the response map, prediction density map, prediction angle map, etc.

[0107] 4. Candidate box generation: Find the local maximum response from the predicted density map to generate candidate target boxes.

[0108] 5. Angle and width / height reasoning: Combine direction prediction with the width and height features of the prompt box to determine the complete parameters of the target box.

[0109] 6. Similarity filtering: Based on semantic space similarity, low-confidence boxes that do not meet the requirements are filtered out.

[0110] 7. Output and Counting: The final output shows the target bounding boxes that have passed through different rotation angles with varying confidence levels, and the count is automatically tallied. This process is real-time and robust, making it particularly suitable for deployment at edge industrial terminals.

[0111] Regarding the construction of the dataset in this application, two data sources are used: Synthetic datasets: Randomly embed objects with transparent backgrounds into complex background images, control the density and angular distribution of the targets, and quickly generate labeled data.

[0112] Self-collected and self-labeled dataset: Collect real industrial scene images, label them with rotating bounding boxes, and define the classification only within the current image, effectively simulating the situation in reality where there is "no global class" but "there is a local class".

[0113] During sampling, for each image, one of all classes contained in the image is randomly sampled to obtain all rotated rectangles of that class, and one is randomly sampled to obtain the example input box.

[0114] The target detection method provided in this application embodiment will be specifically illustrated below through a concrete example: Reasoning phase: The current goal is to demonstrate an apple, so the apple detection process is as follows: 1. Extraction of original features of apples.

[0115] The model takes an image and a tooltip (apple) as input. The image is an RGB image of (1, 3, 512, 512) as input, and the tooltip has a center point of (300, 300), width and height of (100, 50), and a rotation angle of 0°. This tooltip frames an apple. EfficientViT is used as the backbone network to obtain a feature map f with features of (1, 768, 64, 64). Then, a rotated ROI feature is constructed. This can be understood as averaging the portion of feature map f framed by the rotated rectangle, resulting in the original feature of (1, 768). The width and height features are then encoded. At this point, (100, 50) is directly input into a simple neural network to project (1, 768), followed by full-image pooling. Feature map f is globally pooled (averaged) to obtain feature (1, 768), which is then concatenated to form the feature of (3, 768). This is the original feature. During this process, the width and height (100, 50) are also preserved.

[0116] 2. Use the original features to predict possible apples.

[0117] The input image is first processed by EfficientViT as the backbone network to obtain a new feature f1. The previously extracted original features (3, 768) and width / height (100, 50) are also preserved. The image then enters the feature guidance and fusion module. The first step here is to perform a cross-attention query on the original feature (3, 768) using the new feature f1 (1, 768, 64, 64), resulting in the first-level attention mechanism result (1, 768, 64, 64). This first-level attention mechanism result undergoes several self-attention operations to obtain the second-level attention mechanism result (1, 768, 64, 64), which is referred to as the feature response map Q. The feature response map Q then passes through the decoder network, ultimately yielding a prediction density map with the shape (1, 1, 64, 64) and a prediction angle map (1, 1, 64, 64).

[0118] The next step is to analyze the predicted density map and predicted angle map to extract all predicted target objects. Let's assume we get bounding boxes (6, 5) for six target objects. We then extract the rotational ROI features of each bounding box, resulting in six bounding box features (6, 768). We directly calculate the similarity between these six boxes and the feature of the prompt box. For example, if five boxes have a similarity of 0.95 and the other has a similarity of 0.8, we can define a similarity threshold, such as 0.9. Similarities above this threshold are considered correct, and similarities below the threshold are considered incorrect. This yields five bounding boxes, and the target objects selected by these five boxes are the detection results. Finally, we only need to save the original features and dimensions of the apple to detect any target object.

[0119] Training phase: First, a large dataset of top-down images is constructed to extract a single image and obtain all bounding boxes of an object within it. Then, an object is randomly selected as a reference bounding box. Using the original features of this reference bounding box, the predicted density map and predicted angle map are obtained through a process similar to inference. Since the image contains all true bounding boxes, the true density map and angle regression map can be generated from these true bounding boxes. The gradient descent algorithm is then used to optimize the model, adjusting the model parameters to make the predicted density and angle maps as consistent as possible with the true density and angle, while minimizing the similarity of features between correct and incorrect predictions. This results in a model capable of successful inference.

[0120] This application utilizes a small number of prompt boxes to define the prototype of the target of interest. Through the interaction between the prototype and image features, the model is guided to perceive regions similar to the prototype, thereby achieving efficient detection and accurate counting of any target. Compared to traditional detection algorithms, this application does not rely on complex data augmentation or a large number of labeled samples, and can quickly adapt to new target categories during the inference stage.

[0121] It should be noted that the target object detection method provided in this application embodiment can be executed by a target object detection device or a control module within that device for executing the target object detection method. This application embodiment uses the execution of the target object detection method by a target object detection device as an example to illustrate the target object detection device provided in this application embodiment.

[0122] Figure 2 This is a schematic diagram of the structure of a target object detection device according to an embodiment of this application. Figure 2 As shown, the target object detection device 200 includes: an acquisition module 210, an interaction module 220, a generation module 230, and a detection module 240.

[0123] The acquisition module 210 is used to acquire the original features of the reference object pre-marked in the original image based on the feature map of the input original image; the interaction module 220 is used to interact with the feature map to obtain a feature response map; the generation module 230 is used to generate a prediction density map and a prediction angle map based on the feature response map; the detection module 240 is used to mark the target object corresponding to the reference object based on the confidence of each first pixel in the prediction density map as the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features.

[0124] In one embodiment, the acquisition module 210 is configured to acquire the first position rotation feature and the width and height feature of the reference object from the feature map; perform average pooling operation on the feature map to obtain the full image background feature; and combine the first position rotation feature, the width and height feature and the full image background feature into the original feature.

[0125] In one embodiment, the interaction module 220 is configured to use the original features as keys and values, and the feature map as a query for feature update to generate an initial response map, wherein each position of the initial response map includes relevance or difference with the target object; and to perform self-attention enhancement on the initial response map to enhance the relevance or difference between each position of the initial response map and the target object, thereby obtaining the feature response map.

[0126] In one embodiment, the generation module 230 is used to process the feature response map by bilinear interpolation upsampling and convolution to generate the prediction density map and the prediction angle map. The density value of each first pixel in the prediction density map is used to represent the confidence that the first pixel is the center point of the target object. Each third pixel in the prediction angle map includes the corresponding angular direction.

[0127] In one embodiment, the detection module 240 is configured to determine the center point of the target object based on the confidence level of the first pixel in the prediction density map; determine the prediction angle of the center point from the prediction angle map; and mark the target object based on the center point, the prediction angle of the center point, and the width and height features.

[0128] In one embodiment, the detection module 240 is further configured to acquire a first position rotation feature of the reference object; acquire a second position rotation feature of the target object; calculate the similarity between the first position rotation feature and the second position rotation feature; and determine that the reference object and the target object have a corresponding relationship if the similarity is greater than a similarity threshold.

[0129] In one embodiment, the acquisition module 210 is further configured to input the original image into a lightweight neural network backbone structure for feature extraction to obtain the feature map.

[0130] The detection device for the target object in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0131] The target object detection device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0132] The target object detection device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0133] Optionally, such as Figure 3 As shown in the illustration, this application embodiment also provides an electronic device 300, including a processor 301 and a memory 302. The memory 302 stores a program or instructions that can run on the processor 301. When the program or instructions are executed by the processor 301, they perform the following: based on the feature map of the input original image, obtain the original features of a reference object pre-marked in the original image; interact the original features with the feature map to obtain a feature response map; generate a prediction density map and a prediction angle map based on the feature response map; and mark the target object corresponding to the reference object based on the confidence that each first pixel in the prediction density map is the center point of the target object, the prediction angle represented by each second pixel in the prediction angle map, and the width and height features in the original features.

[0134] In one embodiment, the first position rotation feature and the width and height feature of the reference object are obtained from the feature map; the feature map is subjected to average pooling to obtain the full image background feature; the first position rotation feature, the width and height feature and the full image background feature are combined to form the original feature.

[0135] In one embodiment, the original features are used as keys and values, and the feature map is used as a query for feature update to generate an initial response map. Each position of the initial response map includes relevance or difference with the target object. Self-attention enhancement is performed on the initial response map to enhance the relevance or difference between each position of the initial response map and the target object, thereby obtaining the feature response map.

[0136] In one embodiment, the feature response map is processed by bilinear interpolation upsampling and convolution to generate the prediction density map and the prediction angle map. The density value of each first pixel in the prediction density map is used to represent the confidence that the first pixel is the center point of the target object. Each third pixel in the prediction angle map includes the corresponding angular direction.

[0137] In one embodiment, the center point of the target object is determined based on the confidence level of the first pixel in the predicted density map; the predicted angle of the center point is determined from the predicted angle map; and the target object is marked based on the center point, the predicted angle of the center point, and the width and height features.

[0138] In one embodiment, after marking the target object corresponding to the reference object, a first position rotation feature of the reference object is obtained; a second position rotation feature of the target object is obtained; the similarity between the first position rotation feature and the second position rotation feature is calculated; if the similarity is greater than a similarity threshold, it is determined that the reference object and the target object have a corresponding relationship.

[0139] In one embodiment, before obtaining the original features of a pre-labeled reference object in the original image based on the feature map of the input original image, the original image is input into a lightweight neural network backbone structure for feature extraction to obtain the feature map.

[0140] The specific execution steps can be found in the various steps of the above-described target object detection method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0141] It should be noted that the electronic devices in the embodiments of this application include: servers, terminals, or other devices besides terminals.

[0142] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0143] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0144] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0145] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described target object detection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0146] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as ROM, RAM, magnetic disk, or optical disk.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] It should be understood that the training and prediction processes of the models involved in the various embodiments of this specification all adhere to multiple legal and compliant principles, including legal data sources, compliant data content, compliant data governance, compliant training objectives and schemes, compliant training processes, compliant training environments and tools, and compliant ethical verification of training results, and comply with the requirements of Article 5 of the Patent Law. Among them: Data source legitimacy: All datasets used for model training were obtained through legal means, covering three categories: publicly authorized data, data authorized by partners, and self-collected compliant data. Publicly authorized data comes from compliant data sources following open-source licenses such as Apache 2.0, with complete copyright attribution and authorization scope clearly marked, and no unauthorized open-source code or data reuse. Data authorized by partners has been subject to formal data usage agreements, clearly defining the scope, duration, and confidentiality obligations, and possessing a complete authorization chain. For self-collected data involving personal information, strict informed consent procedures have been followed, and anonymization processes (including but not limited to field masking, feature anonymization, and differential privacy technology) have been used to remove personally identifiable information, fully complying with the requirements of the "Interim Measures for the Administration of Generative Artificial Intelligence Services," the "Personal Information Protection Law," and other relevant laws and regulations.

[0150] Data content compliance: The model's dataset undergoes multiple screening and cleaning processes to remove all content that may violate social morality or harm public interests. It contains no obscene, pornographic, violent, discriminatory, or information that endangers national or public safety, nor does it involve the illegal acquisition or use of genetic resources. For data in sensitive fields (such as healthcare and finance), an additional privacy-preserving computation module (including federated learning and secure multi-party computation technologies) ensures that the data is "usable but not visible," avoiding compliance risks during the original data transmission process and ensuring that the data application scenarios and uses comply with public order and good morals and industry regulatory requirements.

[0151] Data governance norms: A complete data traceability system is established during model training to automatically record the source, collection time, annotation process, cleaning rules, and permission allocation of training data, generating traceable compliance reports to ensure data verifiability throughout its entire lifecycle. The model's dataset annotation process is completed by a professional human R&D team, clearly defining the proportion of human creative contribution and avoiding reliance on generated data that has not undergone substantial human modification, thus meeting the "human subject contribution" examination requirements in patent applications.

[0152] Training objectives and plans are compliant: The model training objective focuses on [specific technical scenarios can be supplemented, such as intelligent driving decision optimization, multimodal information interaction, etc., and replaced based on specific content]. The training scheme and the final output results do not violate any mandatory provisions of laws and administrative regulations, do not harm the public interest or the legitimate rights and interests of others, and do not pose any potential risks of being used for illegal activities, infringing on privacy, or undermining public safety. The model strictly adheres to the ethical principle of "intelligent for good".

[0153] Training process compliance: A closed-loop training framework is adopted to ensure compliance and controllability of the training process. The specific process is as follows: First, training samples are obtained through compliant data sources. After the aforementioned data cleaning and desensitization, they are input into the neural network model to generate preliminary training results. Second, an expert system is introduced to verify the preliminary results. Based on preset rules and human expert experience, the feasibility of the results is evaluated, and outputs that may pose ethical risks or compliance hazards are corrected (such as removing decision-making logic that violates public order and good morals, and adjusting model parameters that do not comply with safety regulations). Finally, the loss function weights are dynamically optimized based on expert system feedback to strengthen the model's learning of compliant results, avoid overfitting errors or non-compliant labels, and form a closed-loop control of "data input - model training - expert verification - parameter optimization - result feedback" to ensure that the entire training process complies with A5 ethical review requirements.

[0154] Training environment and tool compliance: Model training is implemented using nationally licensed chips and a compliant training platform. All open-source frameworks and components used in the training process have obtained their corresponding licenses, and copyright statements and patent citation information are fully retained, with no instances of infringement or reuse. The training environment is built using virtual devices (containers / virtual machines) with fixed random seeds and initial parameter configurations to ensure the reproducibility of the training process. At the same time, through access control and operation log recording, risks such as data leakage and parameter tampering during training are prevented, ensuring the security and compliance of the training process.

[0155] Training results ethical verification compliance: After the model is trained, it undergoes additional third-party ethical compliance assessment and algorithm filing review to verify that the model output does not violate social morality or harm public interests. For potentially sensitive scenarios (such as public services and intelligent decision-making), a special result verification mechanism is established to ensure that the model always complies with Article 5 of the Patent Law and relevant laws and regulations in practical applications.

[0156] In summary, the data and training process used in the model in this specification strictly comply with the relevant provisions of Article 5 of the Patent Law and the Patent Examination Guidelines (2023 Edition), and there are no violations of laws, social ethics, public interests, or illegal use of genetic resources. Therefore, it fully meets the compliance requirements for patent authorization.

[0157] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for detecting a target object, characterized in that, include: Based on the feature map of the input original image, obtain the original features of the reference object that was pre-marked in the original image; The original features are interacted with the feature map to obtain a feature response map; Based on the feature response map, a prediction density map and a prediction angle map are generated; Based on the confidence level of each first pixel in the predicted density map as the center point of the target object, the predicted angle represented by each second pixel in the predicted angle map, and the width and height features in the original features, the target object corresponding to the reference object is marked.

2. The method according to claim 1, characterized in that, The step of obtaining the original features of the pre-labeled reference object in the original image based on the feature map of the input original image includes: Obtain the first position rotation feature and the width and height features of the reference object from the feature map; The feature map is subjected to average pooling to obtain the background features of the entire image. The first position rotation feature, the width and height feature, and the full image background feature are combined into the original feature.

3. The method according to claim 1, characterized in that, The step of interacting with the original features and the feature map to obtain a feature response map includes: Using the original features as keys and values, and the feature map as a query, feature updates are performed to generate an initial response map. Each position in the initial response map includes the relevance or difference with the target object. The initial response map is enhanced by self-attention enhancement to increase the relevance or difference between each position in the initial response map and the target object, thereby obtaining the feature response map.

4. The method according to claim 1, characterized in that, The step of generating a prediction density map and a prediction angle map based on the feature response map includes: The feature response map is processed by bilinear interpolation upsampling and convolution to generate the prediction density map and the prediction angle map. The density value of each first pixel in the prediction density map is used to represent the confidence that the first pixel is the center point of the target object. Each third pixel in the prediction angle map includes the corresponding angular direction.

5. The method according to claim 1, characterized in that, The method of marking the target object corresponding to the reference object based on the confidence of each first pixel in the predicted density map as the center point of the target object, the predicted angle represented by each second pixel in the predicted angle map, and the width and height features in the original features includes: The center point of the target object is determined based on the confidence level of the first pixel in the predicted density map; Determine the predicted angle of the center point from the predicted angle diagram; The target object is marked based on the center point, the predicted angle of the center point, and the width and height features.

6. The method according to claim 1, characterized in that, After marking the target object corresponding to the reference object, the method further includes: Obtain the first position rotation feature of the reference object; Obtain the second position rotation feature of the target object; Calculate the similarity between the first position rotation feature and the second position rotation feature; If the similarity is greater than the similarity threshold, it is determined that there is a corresponding relationship between the reference object and the target object.

7. The method according to claim 1, characterized in that, Before obtaining the original features of the pre-labeled reference object in the original image based on the feature map of the input original image, the method further includes: The original image is input into a lightweight neural network backbone structure for feature extraction to obtain the feature map.

8. A target object detection device, characterized in that, include: The acquisition module is used to acquire the original features of a pre-marked reference object in the original image based on the feature map of the input original image; The interaction module is used to exchange information between the original features and the feature map to obtain a feature response map; The generation module is used to generate a prediction density map and a prediction angle map based on the feature response map; The detection module is used to mark the target object corresponding to the reference object based on the confidence that each first pixel in the predicted density map is the center point of the target object, the predicted angle represented by each second pixel in the predicted angle map, and the width and height features in the original features.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the target object detection method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the target object detection method as described in any one of claims 1-7.

Images