Image instance detection method and device and storage medium

Abstract

The application discloses an image instance detection method and device and a storage medium, and relates to the technical field of image detection. Firstly, the application performs standardization processing, 3*3 ROI Pooling feature extraction and perspective disturbance enhancement on a multi-perspective template, and simultaneously adopts offline storage. Then, the feature extraction parameters of the query image are adjusted through a template feature branch. The query image features and the template embedding vectors are projected to the same comparison space to generate a dynamic similarity graph, and feature fusion is realized by combining kernel flip weighted aggregation and a learnable fusion structure. Next, three tasks of boundary box prediction, segmentation and center prediction are cooperatively detected, and a weighted multi-task loss function is used to optimize the model training. Finally, the intersection over union and the mask Dice coefficient double-index non-maximum suppression are used for result screening, which improves the accuracy and robustness of the detection result of the image instance detection in a complex scene.

Description

Technical Field

[0001] This application relates to the field of image detection technology, specifically to an image instance detection method, apparatus, and storage medium. Background Technology

[0002] Instance detection, a core task of computer vision, aims to locate and identify specific object instances (rather than general categories) from images. Examples include detecting specific models of mechanical parts in industrial settings and locating target toys in robotic tasks. This technology plays an irreplaceable role in fields such as intelligent interaction and automated production. For instance, in warehouse robot operations, accurately identifying specific package instances is a prerequisite for automated sorting; in augmented reality, locating user-specified physical objects is key to overlaying virtual content.

[0003] In the field of image instance detection, especially in practical applications such as industrial parts inspection and robot grasping, traditional technologies often face the problem of insufficient detection accuracy in complex scenarios. On the one hand, the difference in perspective between the template and the query image, as well as target occlusion, can easily lead to the dilution of effective features by irrelevant backgrounds during feature fusion. On the other hand, spatial structure information is easily lost during feature extraction and fusion, and the utilization rate of multi-view templates is low and the perspective coverage is insufficient, making it difficult to distinguish similar instances in dense scenes. At the same time, the single detection task and single-index post-processing method further reduce the accuracy and practicality of the detection results, failing to meet the requirements of real-world scenarios for the accuracy and robustness of instance detection. Summary of the Invention

[0004] The purpose of this application is to provide an image instance detection method, apparatus and storage medium to solve the problems of low detection accuracy and poor robustness of traditional image instance detection technology in complex scenes.

[0005] To achieve the above objectives, the first aspect of this application provides an image instance detection method, comprising: Obtain the multi-view template of the target instance, standardize the multi-view template and extract feature blocks through 3×3 ROIPooling, generate template embedding vector after viewpoint perturbation enhancement and store it offline; Extract multi-scale features from the query image, and dynamically adjust the feature extraction parameters through template feature branches to optimize the extraction of query image features; The query image features and template embedding vectors are projected into the same contrast space. An initial score map is generated based on spatial correlation. The dynamic similarity map is obtained by fusing instance dimension normalization and spatial dimension normalization. Based on the dynamic similarity map as weights, the template features are aggregated by kernel flip weighting. The template features and query image features are fused through a learnable fusion structure to output enhanced features. Based on the enhanced features, detection accuracy is optimized by using a weighted multi-task loss function through collaborative detection of three tasks: bounding box prediction, segmentation, and center prediction. The detection results are filtered using the intersection-union ratio and mask Dessian coefficient as dual indicators. The detection box and mask are restored to the original image size according to the coordinate mapping relationship, and the instance detection results are output.

[0006] A second aspect of this application provides an image instance detection apparatus, characterized in that it comprises: The preprocessing module is used to obtain multi-view templates of the target instance, standardize the multi-view templates and extract feature blocks through 3×3 ROI Pooling, generate template embedding vectors after viewpoint perturbation enhancement and store them offline. The feature extraction module is used to extract multi-scale features from the query image. It dynamically adjusts the feature extraction parameters through template feature branches to optimize the extraction of query image features. The feature fusion module projects the query image features and template embedding vectors into the same comparison space, generates an initial score map based on spatial correlation, and obtains a dynamic similarity map by fusing instance dimension normalization and spatial dimension normalization. Based on the dynamic similarity map as weights, the template features are aggregated by kernel flipping weighting, and the template features and query image features are fused through a learnable fusion structure to output enhanced features. The detection module is used to perform collaborative detection based on the enhanced features through three tasks: bounding box prediction, segmentation, and center prediction, and to optimize the detection accuracy using a weighted multi-task loss function. The post-processing module is used to filter the detection results using the cross-union ratio and mask Dessian coefficient as dual indicators of non-maximum suppression. Based on the coordinate mapping relationship, the detection box and mask are restored to the original image size, and the instance detection results are output.

[0007] A third aspect of this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a program that can be loaded by a processor and executed by the above-described image instance detection method.

[0008] The beneficial effects of this application are: This application first preserves complete spatial structure information by standardizing multi-view templates, extracting features using 3×3 ROI Pooling, and enhancing viewpoint perturbation. Offline storage is also employed to improve detection and inference efficiency. Next, the feature extraction parameters of the query image are adjusted through template feature branches to reduce the dilution of effective features. Then, the query image features and template embedding vectors are projected into the same contrast space to generate a dynamic similarity map. Feature fusion is achieved by combining kernel-flipped weighted aggregation and a learnable fusion structure, enabling dynamic allocation of fusion weights and precise alignment of spatial positions. Next, a three-task collaborative detection approach involving bounding box prediction, segmentation, and center prediction is adopted, along with a weighted multi-task loss function to optimize model training. This effectively distinguishes similar instances in dense scenes, reduces false positives and false negatives in occluded scenes, and significantly improves instance detection and segmentation accuracy. Finally, a dual-index non-maximum suppression approach using intersection-union ratio (IU) and mask Dessian coefficient is employed for result filtering. This accurately eliminates duplicate predictions, reduces misjudgments caused by single-index filtering, ensures clean and reliable detection results, and improves the accuracy and robustness of image instance detection in complex scenes.

[0009] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an image instance detection method provided in an embodiment of this application; Figure 2 This is a logical schematic diagram of an image instance detection method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an image instance detection device provided in the embodiments of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified. Details are set forth in the following description for illustrative purposes. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid unnecessarily obscuring the description of this application. Therefore, this application is not intended to be limited to the embodiments shown, but rather to be consistent with the broadest scope of the principles and features disclosed herein.

[0013] Figure 1 This is a flowchart illustrating an image instance detection method provided in an embodiment of this application. Figure 1 As shown, the method may include steps 101-105, which will be described in detail below.

[0014] Step 101: Obtain the multi-view template of the target instance, standardize the multi-view template and extract feature blocks through 3×3 ROI Pooling, generate template embedding vectors after viewpoint perturbation enhancement and store them offline.

[0015] Standardization involves unifying the format and size of multi-view templates, eliminating format differences between them, and providing a unified input for subsequent feature extraction. First, the multi-view template corresponding to the target instance is obtained. Then, standardization is performed on this template to unify its input format and basic specifications, eliminating format and size differences between different view templates, and providing a standardized input foundation for stable feature extraction.

[0016] 3×3 ROI Pooling is an operation for feature extraction of the target region of a template. It extracts fixed-size feature blocks without flattening the features, preserving the spatial structure information of the target. Viewpoint perturbation enhancement expands the viewpoint of the template by simulating viewpoint changes in real-world scenes, improving the template's adaptability to different observation angles. After standardization, 3×3 ROI Pooling is used to extract feature blocks from the target region in the template. This extracts regular feature information without disrupting the feature spatial distribution, fully preserving the spatial structure features of the target instance and reducing spatial information loss due to improper feature processing. Subsequently, viewpoint perturbation enhancement is performed on the template to simulate various viewpoint changes that the target may experience in real-world application scenarios, expanding the template's viewpoint coverage and improving its adaptability to different observation angles and application scenarios.

[0017] Based on the standardized and viewpoint-perturbation-enhanced template, a template embedding vector that can represent the core attributes of the target is extracted and generated. Finally, the template embedding vector is stored offline, eliminating the need to repeat the template feature extraction and generation operations in the detection stage, and directly providing stable feature support for subsequent feature matching.

[0018] A standardized template input method preserves target spatial structure information and expands the template's viewpoint coverage, thereby improving the model's adaptability to viewpoint changes and target occlusion from the source. Offline storage of embedded vectors eliminates repetitive calculations during the testing phase, improving inference efficiency and laying the foundation for high-precision detection.

[0019] Step 102: Extract multi-scale features from the query image and dynamically adjust the feature extraction parameters through template feature branches to optimize the extraction of query image features.

[0020] Multi-scale features are features extracted from the query image at different resolutions and semantic levels. They can simultaneously cover both target detail information and global semantic information, preserving both the detailed texture features of the target instance and the overall semantic features of the target, thus comprehensively capturing the effective feature information in the query image. A dedicated template feature branch is also constructed, which dynamically adjusts the relevant parameters for feature extraction from the query image. This template feature branch is a dedicated branch for analyzing template features and can extract the core feature attributes of the template.

[0021] Dynamically adjusting feature extraction parameters adaptively adjusts the feature extraction weights of the query image based on template features, weakening the interference of irrelevant background features. This step can be guided by the target feature attributes of the template to adaptively optimize the feature extraction weights and strategies, weaken the interference of irrelevant background features in the query image, strengthen the response strength of relevant features of the target instance, and make the final extracted query image features more closely match the actual needs of template feature matching.

[0022] By extracting multi-scale features from the query image and dynamically adjusting feature extraction parameters based on the template feature branch, targeted optimization of query image features is achieved. This dynamic adaptive extraction of query image features reduces the dilution of effective features by the background, improves the feature matching degree between the template and the query image, enhances the robustness of feature extraction in complex scenes, and provides high-quality input for subsequent feature fusion.

[0023] Step 103: Project the query image features and template embedding vectors into the same contrast space, generate an initial score map based on spatial correlation, and obtain a dynamic similarity map by fusing instance dimension normalization and spatial dimension normalization. Based on the dynamic similarity map as weights, aggregate template features by kernel flipping weighting, and achieve the fusion of template features and query image features through a learnable fusion structure to output enhanced features.

[0024] Projecting to the same contrast space maps two types of features to the same feature distribution space, eliminating differences in feature distribution and ensuring the accuracy of similarity calculation. First, the multi-scale features extracted from the query image and the previously generated template embedding vector are simultaneously projected to the same contrast space. This unifies the distribution form and measurement standard of the two types of features, eliminates calculation biases caused by differences in feature distribution, and ensures the accuracy and rationality of subsequent feature similarity calculations.

[0025] Spatial relevance is calculated positionally to determine the degree of matching between the query image and template features, used to characterize the similarity of target regions. After unifying the feature space, the spatial relevance between the query image features and the template embedding vector is calculated positionally to obtain an initial score map representing the degree of feature matching. The initial score map is then normalized in both the instance dimension and spatial dimension to balance the feature contributions of different templates and spatial locations. The two normalization results are then fused to obtain a dynamic similarity map. The dynamic similarity map is a weighted map based on dual-dimensional normalization, which can dynamically distinguish the feature contributions of different locations and templates.

[0026] Kernel-flipped weighted aggregation ensures spatial alignment between template features and the query image through a flipping operation, and then aggregates features using similarity weights. This dynamic similarity map serves as the core weight for feature fusion, and the kernel-flipping operation guarantees precise spatial alignment between template features and the query image, thus completing the weighted aggregation of template features.

[0027] Finally, a learnable fusion structure is used to achieve deep interactive fusion of template features and query image features. This learnable fusion structure is a learning-based structure used to fuse two types of features, preserving the original feature information during the fusion process. This preserves the effective information of the original features and strengthens target-related features, ultimately outputting enhanced features with significantly improved feature quality.

[0028] Through the above steps, dynamic weighted fusion and precise spatial alignment of features can be achieved, which strengthens the effective features of the target, suppresses invalid noise, alleviates the feature dilution problem in occlusion and viewpoint difference scenarios from the root, preserves the spatial structure of the target, and greatly improves the accuracy of feature fusion.

[0029] Step 104: Based on enhanced features, detection is performed through three collaborative tasks: bounding box prediction, segmentation, and center prediction. The detection accuracy is optimized using a weighted multi-task loss function.

[0030] Target instance detection is performed based on the enhanced features obtained from previous fusion, employing a detection mode that coordinates three tasks: bounding box prediction, pixel-level segmentation, and target center prediction. Bounding box prediction enables target instance localization, pixel-level segmentation divides the target into pixel-level regions, and target center prediction verifies and calibrates the target location. These three tasks complement and verify each other, forming a complete closed-loop detection mechanism that comprehensively improves the ability to detect and recognize target instances.

[0031] Simultaneously, a weighted multi-task loss function is constructed to balance and constrain the losses of the three tasks of bounding box prediction, segmentation, and center prediction. This reduces the dominance of single-task loss in the model training process, allowing the three detection tasks to be optimized and improved in synergy, thus comprehensively enhancing the model's detection performance in complex scenarios such as dense targets and target occlusion.

[0032] Three-task collaborative detection is a comprehensive detection mode that integrates localization, segmentation, and center verification. The weighted multi-task loss function is the core constraint method for balancing multi-task training and achieving collaborative optimization. Through the above operations, similar instances in dense scenes can be accurately distinguished, effectively reducing the probability of false detection and false negative detection in occlusion and complex scenes, directly improving the overall accuracy of image instance detection, and allowing the model to maintain stable detection performance in complex application scenarios.

[0033] Step 105: Use the intersection-union ratio and mask Dessian coefficient as dual indicators to filter the detection results. Based on the coordinate mapping relationship, restore the detection box and mask to the original image size and output the instance detection results.

[0034] This step performs post-processing optimization on all detection results output from multi-template, multi-task collaborative detection. A non-maximum suppression method combining intersection-union ratio (IU) and mask Dessian coefficient is used to filter detection results. Simultaneously, duplicate prediction results are determined based on the degree of bounding box overlap and mask overlap, accurately eliminating invalid duplicate detection results and reducing misjudgment issues caused by single-index filtering.

[0035] During the image preprocessing stage, the coordinate mapping relationship between the original image and the standardized image is recorded. After the detection results are filtered, the coordinates of the detection boxes and mask pixels generated on the standardized image are reverse-converted according to this coordinate mapping relationship to restore the size and spatial position of the original image, ensuring spatial consistency between the detection results and the original image. The final output is a complete instance detection result containing instance identifiers, category information, precise bounding boxes, and pixel-level masks, ensuring that the detection results meet the accurate localization requirements of practical applications.

[0036] Dual-index nonmaximum suppression is a precise deduplication method that combines intersection-union ratio (IU) and mask Dessian coefficient. Coordinate mapping is the core basis for restoring the size of the result, and the mask Dessian coefficient is a key indicator for measuring the degree of mask overlap and improving the accuracy of distinguishing dense instances. Through the above operations, the purity and reliability of the detection results are guaranteed, making the detection results adaptable to various practical application scenarios and maintaining high accuracy and robustness in complex scenarios.

[0037] This application first standardizes multi-view templates, extracts features using 3×3 ROI Pooling, and enhances viewpoint perturbation to preserve complete spatial structure information. Offline storage is employed to improve detection and inference efficiency. Then, the feature extraction parameters of the query image are adjusted through template feature branches to reduce the dilution of effective features. Next, the query image features and template embedding vectors are projected into the same contrast space to generate a dynamic similarity map. Feature fusion is achieved by combining kernel-flipped weighted aggregation and a learnable fusion structure, enabling dynamic allocation of fusion weights and precise alignment of spatial positions. Subsequently, a three-task collaborative detection approach involving bounding box prediction, segmentation, and center prediction is adopted, along with a weighted multi-task loss function to optimize model training. This effectively distinguishes similar instances in dense scenes, reduces false positives and false negatives in occluded scenes, and significantly improves instance detection and segmentation accuracy. Finally, a dual-index non-maximum suppression approach using intersection-union ratio (IU) and mask Dessian coefficient is employed for result filtering. This accurately eliminates duplicate predictions, reduces misjudgments from single-index filtering, ensures clean and reliable detection results, and improves the accuracy and robustness of image instance detection in complex scenes.

[0038] In step 101, the multi-view template can first be uniformly converted into a 4-channel format consisting of RGB three-channel and single-channel binary masks. The target region in the multi-view template is marked using the single-channel binary mask. The 4-channel template format is composed of RGB three-channel image information and a single-channel binary mask. The binary mask is used to specifically mark the target instance region, achieving accurate segmentation between the target and the background. The target region is the instance region.

[0039] After format standardization, a bilinear interpolation algorithm is used to normalize the size of the multi-view templates, scaling them to a preset fixed size to ensure consistent input specifications across different view templates. Based on this, for the target region within the multi-view templates, 3×3 ROI Pooling is used to extract feature blocks with dimensions K×C×3×3, where K is the number of templates and C is the number of feature channels, thus preserving the spatial structure information of the target region while fixing the feature size.

[0040] During the template rendering stage, a viewpoint perturbation enhancement operation is performed on the processed template. Random rotation perturbations are added around the X, Y, and Z axes. Multiple uniform viewpoints are sampled on the icosahedron through offline rendering, and each viewpoint is subjected to in-plane rotation. This generates a local template that can cover multiple observation angles, thus comprehensively expanding the viewpoint coverage of the template.

[0041] The standardized and viewpoint-enhanced local templates are input into a template aggregation guide branch built using a pre-trained SqueezeNet-18 architecture. This branch generates a 1×1 resolution global semantic embedding vector and a 3×3 resolution local spatial structure embedding vector, which together form the template embedding vector. The dimension of the template embedding vector matches the number of feature channels in the query image, ensuring compatibility for subsequent feature matching and fusion. The template aggregation guide branch aggregates the semantic and spatial structure features of the template. The global semantic embedding vector (1×1 resolution) represents the semantic feature vector of the overall category and attributes of the target instance. The local spatial structure embedding vector (3×3 resolution) represents the structural feature vector of the local morphology and spatial layout of the target instance.

[0042] Finally, the global semantic embedding vector and the local spatial structure embedding vector are stored locally to achieve offline storage of template embedding vectors. By storing them uniformly locally, the offline storage of template embedding vectors is completed, and they can be directly called in subsequent detection processes without the need for repeated generation.

[0043] In step 102, feature extraction and optimization are performed on the query image to be detected. First, a DenseNet-121 network pre-trained on ImageNet is used as the backbone network. DenseNet-121 is a deep convolutional neural network with a densely connected structure. It improves feature extraction efficiency and representation ability through dense reuse of feature maps. Pre-training on the ImageNet dataset gives it a stronger foundation for general feature extraction. Leveraging the general feature extraction capabilities learned by this network on large-scale image datasets, a high-quality basic model is provided for query image feature extraction. An adjustable filter is inserted after the first layer of the backbone network. This adjustable filter is a dynamic weight filter inserted after the first layer of the backbone network. Its parameters are output by the template feature branch and can be adaptively adjusted according to the characteristics of the target instance, achieving targeted optimization of feature extraction. This adjustable filter acts as a dynamic adjustment switch for feature extraction, used to subsequently adaptively optimize the weights of lower-level feature extraction based on template features, enabling the backbone network to optimize lower-level feature extraction according to the characteristics of the target instance (e.g., enhancing the red channel feature response when detecting red parts).

[0044] A feature pyramid network is connected after the backbone network. Through this network's multi-scale feature fusion mechanism, multiple feature maps at different resolution levels are generated. The feature pyramid network is a network architecture for generating multi-scale feature maps. It fuses features from different levels through top-down feature propagation and lateral connections, taking into account both target details and global semantic information. Lower-level feature maps preserve details such as edges and textures of the target instance, while higher-level feature maps enhance the target's global semantic information, achieving full coverage of multi-level features from details to semantics.

[0045] Taking a feature map with 5 levels as an example, the feature levels are P2 to P6, and the characteristics and functions of each level of feature map are shown in Table 1.

[0046] Table 1 The process of feature fusion and channel adjustment includes: adjusting the feature maps of each level of the feature pyramid network to 256 channels through 1×1 convolution, then upsampling the low-level feature maps (P2, P3) to the same resolution (80×80) as P4 through bilinear interpolation, and finally stitching them along the channel dimension to generate the comprehensive features of the query image.

[0047] To eliminate the impact of channel differences between feature maps at different levels on subsequent fusion, a 1×1 convolution is used to uniformly adjust the feature maps of each level to the same number of channels. The low-level feature maps are then upsampled to a uniform resolution and concatenated with other feature maps along the channel dimension to generate multi-scale features of the query image. For example, for low-resolution low-level feature maps, bilinear interpolation is used for upsampling to scale them up to the same resolution as other feature maps. Figure 1Achieving a uniform resolution. Bilinear interpolation is an interpolation algorithm used for image / feature map resolution scaling. It can increase the size of the feature map while preserving details such as edges and textures, reducing feature distortion caused by scaling. Then, all hierarchical feature maps are concatenated along the channel dimension to ultimately generate multi-scale features for the query image that combine detailed integrity with semantic richness.

[0048] Simultaneously, a dual-template feature branch is constructed to dynamically guide feature extraction parameters. The first is the global template branch (covering the complete form of the target from a single perspective, such as a front view). This branch uses the SqueezeNet-18 architecture to construct the global template branch, taking a global template covering the complete form of the target instance as input. Through feature processing and parameter learning in this branch, adjustable filter parameters are output (dimensions consistent with the first layer convolutional kernel of the backbone network, 64×3×7×7). These parameters are injected into the first layer of the backbone network via residual connections, dynamically adjusting the weights of the adjustable filter to accurately adapt to the feature attributes of the target instance. The second is the local template branch, which uses the same structure as the global template branch and is trained independently using the SqueezeNet-18 architecture to ensure parameter independence. Taking multi-view local templates as input, it outputs a 1×1 resolution global semantic embedding vector (e.g., the category attribute of "industrial bolt") and a 3×3 resolution local spatial structure embedding vector (e.g., the thread texture and head groove of the bolt), providing dedicated feature support for subsequent feature matching.

[0049] Finally, the adjustable filter parameters output by the global template branch are used to optimize the backbone network's weights for extracting query image features, thereby strengthening the feature responses associated with the target. For example, the response intensity of features such as color, texture, and shape related to the target instance is enhanced, while the interference of irrelevant features such as background and noise in the query image is weakened, so that the final output query image features have stronger target specificity and matching adaptability.

[0050] This step leverages the general feature extraction capabilities and dense connections of ImageNet's DenseNet-121 to lay a high-quality foundation for query image feature extraction, reducing model training costs. The introduction of a feature pyramid network enables comprehensive capture of multi-scale features, solving the problem that single-scale features struggle to handle both large and small targets, as well as occluded targets. By using 1×1 convolutions to unify the number of channels and upsampling low-level features, the compatibility and fusion effectiveness of multi-scale features are ensured, enhancing the comprehensive representational ability of features. The collaborative design of dual-template feature branches provides precise dynamic guidance for feature extraction. The parameters output by the global template branch dynamically adjust the weights of adjustable filters through residual connections, ensuring the feature extraction process accurately adapts to the characteristics of the target instance. The local template branch is trained independently and outputs dual-resolution embedding vectors, further enriching the target feature representation. The combination of these two approaches achieves targeted optimization of query image features, effectively weakening background noise interference, preventing the dilution of effective features, and significantly improving the matching accuracy between template and query image features. The overall process enhances the stability and robustness of feature extraction in complex scenarios through a collaborative mechanism of multi-scale feature capture and template-guided dynamic optimization, providing high-quality input for subsequent feature fusion and detection tasks.

[0051] In step 103, the query image features and template embedding vectors are first projected onto the same contrast space using 1×1 convolution and a shared LayerNorm projection operation. The 1×1 convolution, with a kernel size of 1×1, adjusts the feature distribution and unifies the feature representation without changing the feature dimension or spatial size. The shared LayerNorm is a layer normalization operation with shared parameters, which normalizes the query image features and template embedding vectors, unifying the feature distribution and eliminating distribution bias. The same contrast space maps two types of features from different sources to the same feature metric space, ensuring fairness and accuracy in similarity calculation. By projecting to a unified space and ensuring dimensionality consistency before and after projection, the interference of feature distribution differences on similarity calculation is eliminated, laying the foundation for subsequent accurate feature correlation calculation. Using the projected template embedding vector as the convolution kernel, convolution operations are performed on the projected query image features, calculating the correlation between the query image features and the template embedding vector at each spatial location to generate an initial score map.

[0052] The initial score map is a feature similarity map calculated spatially by convolution operations, which intuitively reflects the degree of feature matching between the query image and the template. The data expression of the initial score map can be: ; in, This represents the initial score graph. This represents the convolution operation. This represents the projection operation of "1×1 convolution + LayerNorm". Indicates querying image features, This represents the template embedding vector. Essentially, this convolution operation calculates the correlation between the query image features and the template features at each spatial location; a higher score indicates a greater similarity between that location and the template target.

[0053] A two-dimensional normalization process is performed on the initial score map. Softmax normalization is applied along the template dimension to highlight template features with higher contribution, while maximum normalization is applied along the spatial dimension to ensure that the similarity scores at each spatial location fall within a preset range, balancing the feature response intensity at different spatial locations. Softmax normalization, or instance-dimensional normalization, is a normalization operation performed along the template quantity dimension to distinguish the feature contribution weights of different templates. Maximum normalization, or spatial-dimensional normalization, is a normalization operation performed along the spatial dimension of the feature map, constraining the similarity scores within a stable range and improving the interpretability of spatial locations.

[0054] In one example, instance dimension normalization (ENorm) is based on ( Along template dimension ( Softmax normalization is performed to highlight the template that contributes the most, using the following formula: ; in, For instance dimension normalization results, , where is the template feature block size, and the normalized denominator is used to balance the influence of feature dimension on the score.

[0055] Spatial Dimension Normalization (SNorm): for ( Along spatial dimensions ( Maximum value normalization is performed to ensure that the similarity score of each spatial location falls within the [0,1] interval, thus ensuring the spatial interpretability of the similarity graph. The formula is as follows: ; in, This is the result of spatial dimension normalization.

[0056] Next, the instance dimension normalization result and the spatial dimension normalization result are fused through element-wise multiplication to generate a dynamic similarity graph. The dynamic similarity graph is a dynamic weighted graph generated by the fusion of two-dimensional normalization, which simultaneously represents the difference in contribution of different templates and the degree of matching of spatial positions.

[0057] Then, the template embedding vector is subjected to kernel flipping in both the horizontal and vertical directions to ensure precise spatial alignment between the template features and the query image. For example, the "left-right" and "top-bottom" positional relationships of the target in the template are consistent with those in the query image. Kernel flipping is an operation that flips the template embedding vector horizontally and vertically to correct spatial positional deviations between the template and the query image, ensuring spatial structure alignment. Experiments show that the flipping operation improves the model's instance discrimination accuracy by 5.2% in dense scenes.

[0058] With dynamic similarity graph Using the flipped template embedding vector as the weight, a convolution operation is performed to generate similarity-weighted template features. Then, sum along the template dimension to obtain the final weighted template features. Similarity-weighted template features are template features obtained by aggregating dynamic similarity maps as weights and flipped template embedding vectors as convolution kernels, thus enhancing the target-related feature response. The formula is as follows: ; This process gives higher template feature weights to regions in the query image that are similar to the template, thereby enhancing the feature response of the target region.

[0059] Then, a learnable fusion structure composed of multi-layer convolution, residual connections, and LayerNorm is adopted. Weighted template features and query image features The fusion process preserves the original, effective features of the query image while eliminating fusion bias caused by differences in feature scale. The learnable fusion structure is a feature fusion module composed of multiple convolutional layers, residual connections, and LayerNorms. It adaptively learns the feature fusion strategy, preserving effective features and optimizing feature interactions. In one example, the convolutional block consists of two 3×3 convolutional layers with Leaky ReLU activation to capture... and The interaction characteristics. Residual connections will combine the features of the original query image. The features are directly superimposed onto the output of the convolutional block, reducing the loss of original feature information during the fusion process. LayerNorm performs layer normalization on the fused features to eliminate feature scale differences and ensure the stability of subsequent detection head inputs. Finally, the enhanced features are output after fusion. Enhanced features are the final features after fusion, while retaining the multi-scale semantic and spatial structure information of the query image and the target feature information of the template, thus greatly improving feature quality and adaptability.

[0060] In step 104, the bounding box prediction task is first performed. Multiple anchor boxes covering various scales and aspect ratios are used. These anchor boxes are pre-defined candidate bounding boxes with different scales and aspect ratios to cover targets of different sizes and shapes, providing an initial reference for bounding box prediction. Furthermore, the size of each anchor box is adaptively adjusted according to the feature hierarchy. For example, lower-level feature maps correspond to smaller anchor boxes to adapt to small target detection, while higher-level feature maps correspond to larger anchor boxes to adapt to large target detection, ensuring that targets of different sizes and shapes are effectively covered. In the category classification stage, Focal Loss is used to classify the foreground and background. For example, Focal Loss is used as the loss function, and by dynamically adjusting sample weights, the class imbalance problem where the number of background anchor boxes far exceeds that of foreground anchor boxes is addressed, strengthening the feature learning of foreground targets. In the coordinate regression stage, Smooth L1 Loss is used to predict the offset of the anchor boxes relative to the true bounding boxes, reducing the interference of outliers on the accuracy of bounding box regression and accurately predicting the offset of the anchor boxes relative to the true bounding boxes.

[0061] Simultaneously, a first set value and a second set value are determined based on the intersection-over-union (IOU) ratio between the anchor boxes and the ground truth bounding boxes, wherein the first set value is greater than the second set value. Anchor boxes with an IOU ratio greater than or equal to the first set value are identified as positive samples, and anchor boxes with an IOU ratio less than the second set value are identified as negative samples, thereby completing the sample partitioning for bounding box prediction and reducing the interference of ambiguous samples on training.

[0062] For example, the anchor frame design uses 24 anchor frames (8 scales: 30, 60, 90, 120, 150, 180, 210, 240 pixels; 3 aspect ratios: 0.5, 1, 2) to cover targets of different sizes and shapes (such as slender industrial parts and square electronic components). The anchor frame size is adaptively adjusted according to the hierarchy of the feature pyramid network (small anchor frames for lower-level feature maps and large anchor frames for higher-level feature maps). The foreground (target) / background probability of each anchor frame is predicted using... To address the category imbalance problem (where the number of background anchor boxes far exceeds the number of foreground anchor boxes), among which, For category classification loss, For focus loss function, This represents the probability value of the foreground (target). For real labels, =1 indicates that the anchor frame is a positive sample (corresponding to the foreground target). =0 indicates a negative sample (background). As a balance factor, For focusing parameters. Predicts the offset of the anchor box relative to the target ground truth bounding box. Smooth L1 Loss is used to reduce the impact of outliers on the regression. For bounding box regression loss, The x-coordinate of the bounding box center. The ordinate of the center of the bounding box. The width of the bounding box. The height of the bounding box. The offset of the anchor box predicted by the model relative to the true bounding box (such as the predicted center offset, width offset, etc.). This represents the true offset (i.e., the actual offset from the anchor box to the ground truth bounding box, used as the label for the regression task). Anchor boxes with an IOU ≥ 0.5 are considered positive samples, and those with an IOU < 0.4 are considered negative samples. Anchor boxes with an IOU between 0.4 and 0.5 are ignored to reduce the interference of ambiguous samples on training.

[0063] Next, the segmentation task is performed. A segmentation network consisting of multiple layers of 3×3 convolutions and multiple bilinear upsampling operations is established. The 3×3 convolutions are used to capture pixel-level feature interaction information. A binary mask with the same size as the original query image (e.g., 640×640) is output through the Sigmoid activation function. This mask can accurately mark the pixel-level regions of the target and clarify the pixel attribution between the target and the background, as well as overlapping targets (e.g., distinguishing the pixel attribution of overlapping targets). As an example, the network structure can consist of 5 layers of 3×3 convolutions (with Leaky ReLU activation function) + 2 bilinear upsampling operations (each upsampling by 2 times) to gradually restore the feature map resolution. Finally, the probability of each pixel belonging to the target is output through the Sigmoid activation function (a threshold of 0.5 is used to determine pixel attribution). The segmentation loss is calculated using binary cross-entropy loss, calculating the loss separately for target pixels and background pixels in the mask, focusing on penalizing segmentation errors in occluded regions to complete the segmentation task and improve segmentation accuracy in occluded scenarios (e.g., missing mask prediction for the occluded part of the target).

[0064] Subsequently, the center prediction task is performed. Enhanced features are compressed into a single-channel heatmap (29×39 resolution, consistent with the high-level feature map resolution of the feature pyramid network) using 1×1 convolutions to predict the spatial location of the target center. This is used to verify the consistency between the bounding box and the mask, and to filter false positives. The predicted heatmap is then activated by a Sigmoid function. The true heatmap is constructed with a two-dimensional Gaussian distribution based on the target center location (e.g., a Gaussian distribution with amplitude 1 and variance 5 at the target center). The probability value is highest in the center region and gradually decreases with increasing distance, forming a true label with spatial gradient. L1 loss is used to calculate the difference between the predicted heatmap and the true heatmap, forcing the model to accurately learn the spatial location features of the target center, assisting in bounding box coordinate correction, improving the accuracy of target localization, and thus completing the center prediction task.

[0065] Finally, a total loss function is constructed, which is a weighted average of segmentation loss, center prediction loss, classification loss, and bounding box regression loss. The formula for the total loss function is as follows: ; in, To divide the loss weights, Loss weights are centered. For Focal Loss, The algorithm uses Smooth L1 Loss. The weight settings were experimentally verified to ensure that segmentation and center prediction tasks effectively assist bounding box detection, while reducing the dominance of any single task's loss during training.

[0066] By balancing the loss proportions of each task through weighting coefficients, the model training process is reduced from being dominated by the loss of a single task. The segmentation task clarifies the pixel attribution of overlapping targets, defining the boundaries of overlapping targets in dense scenes. The center prediction task filters false detection results in the background region, filtering out false detection boxes in the background area. The bounding box prediction task, aided by the first two tasks, achieves accurate target localization. These three tasks work together to form a closed-loop detection mechanism of "localization-segmentation-verification," achieving synergy among the three tasks and comprehensively optimizing detection accuracy.

[0067] In step 105, all detection results output from multiple templates and tasks are first sorted in descending order of category confidence. A predetermined number of detection results with the highest confidence are retained as the results to be screened. Category confidence is the probability value that the model determines a detection result belongs to the target category. It is used to characterize the reliability of the detection results and is the core basis for sorting and screening. In this way, computational redundancy in subsequent screening is reduced, while ensuring that core high-confidence results are not missed.

[0068] Subsequently, dual-index non-maximum suppression (NMS) filtering is performed on all the detection results to be screened, calculating the intersection-union ratio (CUI) and mask Desseaucoefficient (MDE) of the bounding boxes between the detection results. When the CUI of any two detection results reaches a set CUI and the MDE reaches a set coefficient, it is judged as a duplicate prediction, and the detection result with higher confidence is retained, reducing the misjudgment problem caused by single-index filtering. The CUI is the ratio of the area of ​​the overlapping region of two bounding boxes to the area of ​​their merged region, used to measure the spatial overlap of the bounding boxes, and is one of the key indicators for determining duplicate predictions. The MDE measures the pixel-level overlap between two masks. By calculating the ratio of the intersection to the union of the masks, it accurately reflects the pixel-level overlap of the target instance, and is a core supplement to dual-index filtering. Dual-index NMS combines the bounding box CUI and MDE to eliminate duplicate results, taking into account both spatial location and pixel attribution, and is more accurate in distinguishing duplicate predictions than single-index NMS. For example, when IOU > 0.5 and the mask Dessian coefficient > 0.6, it is judged as a duplicate prediction, and the result with higher confidence is retained. Compared with the traditional single IOU index non-maximum suppression screening, the dual index can more accurately distinguish duplicate instances (such as when the bounding boxes of two adjacent objects have a high IOU, but the masks do not overlap, they will not be misjudged as duplicates), and the accuracy of duplicate prediction removal is improved by 12.5% ​​in dense scenes.

[0069] In the query image preprocessing stage, the coordinate mapping relationship between the original image and the standardized image is recorded. This coordinate mapping relationship, recorded during query image preprocessing, involves the size and coordinate transformation between the original and standardized images, used to restore the detection results from the standardized space to the original space. For example, the coordinate mapping relationship can include key information such as size scaling ratio and coordinate offset. During the query image preprocessing stage (e.g., scaling to 640×640), the mapping relationship between "original image coordinates and standardized image coordinates" is recorded (e.g., if the original image size is 1280×720, after scaling it to 640×640, the mapping ratio is 0.5). After the detection results are filtered, the detection box coordinates and mask pixel coordinates on the standardized image are reverse-converted according to the coordinate mapping relationship to restore the original image size. The output includes instance identifiers, category information, precise bounding boxes, and pixel-level masks. The spatial location of the detection results is consistent with the original image. The instance identifier distinguishes different target instances, the category information clarifies the target's category, the precise bounding box provides target location information, and the pixel-level mask clarifies the target's pixel-level region, ensuring both accuracy and practicality in the detection results.

[0070] Figure 2 This is a logical schematic diagram of an image instance detection method provided in an embodiment of this application. Figure 2As shown, this image example detection method involves a two-branch collaborative approach of offline template preprocessing and online monitoring inference. Through the full-link logic of template feature pre-generation → dynamic extraction of query features → similarity-guided fusion → three-task closed-loop detection, it fundamentally solves the core problems of low detection accuracy and poor robustness of traditional technologies in occlusion, viewpoint differences and dense scenes.

[0071] For the offline template preprocessing branch, starting with the object physics model of the target instance, multi-view template images are generated through offline rendering: uniform viewpoints are sampled on a regular icosahedron, and in-plane rotation and random rotation perturbations are added to simulate various viewpoint changes of the target in the actual scene, generating local templates covering multiple viewpoints to solve the problem of insufficient template viewpoint coverage. Then, the template images are standardized: uniformly converted to a 4-channel format of RGB three-channel + single-channel binary mask, and the target region is accurately marked by the mask; then, 3×3 ROI Pooling is used to extract feature blocks of the target region, fully preserving spatial structure information and reducing spatial structure loss. Guided by the template aggregation branch of the pre-trained SqueezeNet-18 architecture, feature aggregation is performed on the multi-view local templates to generate 1×1 global semantic embedding vectors (representing target category attributes) and 3×3 local spatial structure embedding vectors (representing target spatial layout), which are then stored offline. Pre-generating and storing template embedding vectors eliminates the overhead of repetitive feature calculations during the detection stage. At the same time, multi-view rendering expands the template coverage, improving the model's adaptability to changes in viewpoint and target occlusion scenarios, and providing stable target feature support for online detection.

[0072] For the online detection branch, multi-scale feature extraction of the query image is performed first. The query image to be detected is first input into the backbone network, such as DenseNet-121, with an adjustable filter inserted after the first layer. A feature pyramid network generates feature maps at multiple resolution levels, with lower-level features preserving detail and texture, and higher-level features enhancing global semantics. A 1×1 convolution unifies the number of feature channels at each level, upsampling the lower-level feature maps to a uniform resolution before concatenating them with other levels to form the multi-scale features of the query image. Simultaneously, the global template branch (SqueezeNet-18 architecture) takes the global template as input and outputs adjustable filter parameters, which are injected into the first layer of the backbone network through residual connections. This dynamically adjusts the filter weights, ensuring the feature extraction process accurately adapts to the characteristics of the target instance and weakens background interference. In this way, targeted feature extraction of the query image is achieved, strengthening the response of target-related features, reducing the dilution of effective features by background noise, and improving the matching accuracy between the template and the query image features.

[0073] Next, similarity comparison is performed to generate dynamic weights. The query image features and offline-generated global / local template embedding vectors are projected onto the same contrast space through 1×1 convolution and a shared LayerNorm to eliminate feature distribution differences. Using the projected template embedding vectors as convolution kernels, convolution operations are performed on the query image features, calculating the correlation at each spatial location to generate an initial score map. Softmax normalization is performed on the initial score map along the template dimension (highlighting high-contribution templates) and maximum value normalization is performed along the spatial dimension (constraining the similarity score interval). The two normalization results are then fused through element-wise multiplication to generate a dynamic similarity map. This map simultaneously represents the contribution differences of different templates and the degree of matching at spatial locations, providing a dynamic weight basis for subsequent fusion. This solves the problem of similarity calculation bias caused by inconsistent feature distributions from different sources, achieves adaptive allocation of fusion weights, and reduces feature dilution caused by fixed weights.

[0074] Then, dynamic weighted feature enhancement is performed. Kernel flipping operations are performed horizontally and vertically on the template embedding vector to ensure precise spatial alignment between the template features and the query image. Using the dynamic similarity map as weights, the flipped template embedding vector is used as a convolution kernel to generate similarity-weighted template features, which are then summed along the template dimension to obtain the final weighted template features. A learnable fusion structure composed of multi-layer convolutions, residual connections, and LayerNorm is used to deeply fuse the weighted template features and query image features. This eliminates feature scale differences while preserving the original effective features, ultimately outputting enhanced features that retain the multi-scale semantics and spatial structure information of the query image, as well as the target-specific features of the template. This achieves precise spatial alignment and dynamic weighting of feature fusion, strengthens effective target features, suppresses background noise, and fundamentally alleviates feature dilution problems in occlusion and viewpoint difference scenarios, significantly improving the accuracy and robustness of feature fusion.

[0075] Then, a three-task collaborative detection is performed. Based on enhanced features, a closed-loop detection mechanism is formed through three parallel detection heads. The center prediction head compresses the enhanced features into a single-channel heatmap through 1×1 convolution, outputs the target center prediction result after Sigmoid activation, and uses a two-dimensional Gaussian distribution for the ground truth label. The L1 Loss constraint model accurately locates the target center, assisting in bounding box correction and false detection filtering. The bounding box prediction head uses multi-scale, multi-aspect ratio adaptive anchor boxes, performs foreground / background classification through Focal Loss (solving the class imbalance problem), and predicts the offset of the anchor box relative to the ground truth bounding box through Smooth L1 Loss. Positive and negative samples are divided according to the intersection-union ratio (IU) double threshold to achieve accurate target localization. The segmentation head generates a binary mask with the same size as the original image through multiple 3×3 convolutions and multiple bilinear upsampling. Binary cross-entropy loss is used to penalize segmentation errors in occluded areas, achieving pixel-level differentiation between the target and the background and overlapping targets. We construct a weighted multi-task loss function to balance segmentation loss, center prediction loss, classification loss, and regression loss, reducing the dominance of single task in training and enabling the three tasks to be optimized collaboratively, forming a closed-loop detection mechanism of "localization-segmentation-verification". This accurately distinguishes similar instances in dense scenes, significantly reduces false detection and false negative rates in occluded and complex scenes, and directly improves detection accuracy.

[0076] This architecture expands the field of view coverage through offline template preprocessing, achieves targeted optimization through online feature extraction, strengthens effective features through similarity fusion and dynamic weighting, and forms a closed-loop verification through three-task detection. Finally, through dual-index non-maximum suppression and coordinate restoration in step 105, it outputs high-quality detection results that are consistent with the spatial position of the original image. It solves the core problems of low detection accuracy and poor robustness of existing technologies from the whole link level, and has higher accuracy and scene adaptability in practical scenarios such as industrial parts inspection and robot grasping.

[0077] Figure 3 This is a schematic diagram of the structure of an image instance detection device 300 provided in an embodiment of this application. Figure 3 As shown, the image instance detection device 300 may include a preprocessing module 301, a feature extraction module 302, a feature fusion module 303, a detection module 304, and a postprocessing module 305.

[0078] The preprocessing module 301 is used to obtain multi-view templates of the target instance, standardize the multi-view templates and extract feature blocks through 3×3 ROI Pooling, generate template embedding vectors after viewpoint perturbation enhancement and store them offline.

[0079] The feature extraction module 302 is used to extract multi-scale features of the query image. It dynamically adjusts the feature extraction parameters through template feature branches to optimize the extraction of query image features.

[0080] The feature fusion module 303 is used to project the query image features and template embedding vectors into the same contrast space, generate an initial score map based on spatial correlation, and obtain a dynamic similarity map by fusing instance dimension normalization and spatial dimension normalization. Based on the dynamic similarity map as weights, the template features are aggregated by kernel flipping weighting, and the template features and query image features are fused through a learnable fusion structure to output enhanced features.

[0081] The detection module 304 is used for detection based on enhanced features, through collaborative detection of three tasks: bounding box prediction, segmentation, and center prediction, and optimizes detection accuracy using a weighted multi-task loss function.

[0082] The post-processing module 305 is used to filter the detection results by using the cross-union ratio and mask Dessian coefficient as dual indicators for non-maximum suppression. Based on the coordinate mapping relationship, the detection box and mask are restored to the original image size, and the instance detection results are output.

[0083] The preprocessing module 301, feature extraction module 302, feature fusion module 303, detection module 304, and postprocessing module 305 can be used to execute steps 101-105 in the embodiments of the above image instance detection method. For the specific implementation of these modules and more details, please refer to the corresponding method section, which will not be elaborated here.

[0084] This application also provides a computer-readable storage medium storing a program that can be loaded by a processor and executed as any of the image instance detection methods in this application.

[0085] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0086] The above examples illustrate this application only to aid understanding and are not intended to limit its scope. Those skilled in the art to which this application pertains can make various simple deductions, modifications, or substitutions based on the ideas presented.

Claims

1. An image instance detection method, characterized in that, include: A multi-view template of the target instance is obtained, the multi-view template is standardized and feature blocks are extracted by 3×3 ROIPooling, and the template embedding vector is generated after viewpoint perturbation enhancement and stored offline. Extract multi-scale features from the query image, and dynamically adjust the feature extraction parameters through template feature branches to optimize the extraction of query image features; The query image features and template embedding vectors are projected into the same contrast space. An initial score map is generated based on spatial correlation. The dynamic similarity map is obtained by fusing instance dimension normalization and spatial dimension normalization. Based on the dynamic similarity map as weights, the template features are aggregated by kernel flip weighting. The template features and query image features are fused through a learnable fusion structure to output enhanced features. Based on the enhanced features, detection accuracy is optimized by using a weighted multi-task loss function through collaborative detection of three tasks: bounding box prediction, segmentation, and center prediction. The detection results are filtered using the intersection-union ratio and mask Dessian coefficient as dual indicators. The detection box and mask are restored to the original image size according to the coordinate mapping relationship, and the instance detection results are output.

2. The image instance detection method according to claim 1, characterized in that, The process of obtaining a multi-view template of the target instance, standardizing the multi-view template, and extracting feature blocks using 3×3 ROI Pooling includes: The multi-view template is uniformly converted into a 4-channel format of RGB three-channel and single-channel binary mask, and the target area in the multi-view template is marked by the single-channel binary mask; The multi-view template is scaled to a preset fixed size using a bilinear interpolation algorithm. For the target region in the multi-view template, feature blocks with dimensions of K×C×3×3 are extracted using 3×3 ROI Pooling, where K is the number of templates and C is the number of feature channels.

3. The image instance detection method according to claim 2, characterized in that, The process of generating template embedding vectors after viewpoint perturbation enhancement and storing them offline includes: During the template rendering stage, random rotation perturbations are added around the X, Y, and Z axes. Multiple uniform viewpoints are sampled on the regular icosahedron through offline rendering, and each viewpoint is subjected to in-plane rotation to generate a local template covering multiple viewpoints. The local template is input into the template aggregation bootstrap branch of the pre-trained SqueezeNet-18 architecture; The template aggregation guides the generation of a 1×1 resolution global semantic embedding vector and a 3×3 resolution local spatial structure embedding vector, resulting in a template embedding vector. The dimension of the template embedding vector is consistent with the number of feature channels of the query image. The global semantic embedding vector and the local spatial structure embedding vector are stored locally to achieve offline storage of the template embedding vector.

4. The image instance detection method according to claim 1, characterized in that, The extraction of multi-scale features from the query image, which involves dynamically adjusting feature extraction parameters through template feature branches to optimize feature extraction, includes: A DenseNet-121 pre-trained on ImageNet is used as the backbone network, and an adjustable filter is inserted after the first layer of the backbone network. A feature pyramid network is then connected after the backbone network to generate multiple feature maps at different resolution levels. The feature maps of each level are uniformly adjusted to the same number of channels by 1×1 convolution, and the low-level feature maps are upsampled to a uniform resolution and then concatenated with other feature maps along the channel dimension to generate the multi-scale features of the query image. A global template branch is constructed using the SqueezeNet-18 architecture. The global template is taken as input and the output is an adjustable filter parameter. This parameter is then injected into the first layer of the backbone network through residual connections to dynamically adjust the weights of the adjustable filter. A local template branch, which is trained independently and has the same structure as the global template branch, is used as input. The multi-view local template is used as input, and the output is a global semantic embedding vector with a resolution of 1×1 and a local spatial structure embedding vector with a resolution of 3×3. The adjustable filter parameters are used to optimize the backbone network's extraction weights for the query image features, thereby enhancing the feature response associated with the target.

5. The image instance detection method according to claim 1, characterized in that, The process of projecting the query image features and template embedding vectors into the same comparison space, generating an initial score map based on spatial correlation, and fusing it through instance dimension normalization and spatial dimension normalization to obtain a dynamic similarity map includes: The query image features and the template embedding vector are projected into the same contrast space through 1×1 convolution and projection operation with shared LayerNorm, and the dimensions are consistent before and after projection. Using the projected template embedding vector as the convolution kernel, a convolution operation is performed on the projected query image features, and the correlation between the query image features and the template embedding vector is calculated at each spatial location to generate an initial score map; The initial score map is subjected to Softmax normalization along the template dimension and maximum value normalization along the spatial dimension, so that the similarity score of each spatial location falls within a preset range. The normalized results of the instance dimension and the normalized results of the spatial dimension are fused by element-wise multiplication to generate a dynamic similarity graph. The dynamic similarity graph simultaneously represents the difference in contribution of different templates and the degree of matching of spatial positions.

6. The image instance detection method according to claim 5, characterized in that, The process involves using the dynamic similarity graph as weights, weighted aggregation of template features through kernel flipping, and then fusing the template features with the query image features via a learnable fusion structure to output enhanced features, including: The template embedding vector is flipped horizontally and vertically, and the dynamic similarity map is used as the weight. The flipped template embedding vector is used as the convolution kernel to perform convolution operation to generate similarity-weighted template features. Finally, the weighted template features are obtained by summing along the template dimension. A learnable fusion structure consisting of multi-layer convolution, residual connections, and LayerNorm is used to fuse the weighted template features with the query image features; After fusion, the enhanced features are output, which simultaneously retain the multi-scale semantics and spatial structure information of the query image, as well as the target feature information of the template.

7. The image instance detection method according to claim 1, characterized in that, The method based on the enhanced features, through collaborative detection of three tasks—boundary box prediction, segmentation, and center prediction—and by utilizing a weighted multi-task loss function to optimize detection accuracy, includes: Multiple anchor boxes covering multiple scales and aspect ratios are used, and the size of each anchor box is adaptively adjusted according to the feature level. Focal Loss is used to classify the foreground and background, and Smooth L1 Loss is used to predict the offset regression of the anchor box relative to the true bounding box. A first set value and a second set value are determined according to the intersection-union ratio (IU) between the anchor frame and the true bounding box. Anchor frames with an IU greater than or equal to the first set value are determined as positive samples, and anchor frames with an IU less than the second set value are determined as negative samples, so as to complete the sample division for bounding box prediction. The first set value is greater than the second set value. A segmentation network consisting of multiple 3×3 convolutions and multiple bilinear upsamplings is established. A binary mask with the same size as the original query image is output through the Sigmoid activation function. The segmentation loss is calculated using binary cross-entropy loss, which focuses on penalizing segmentation errors in occluded regions in order to complete the segmentation task. The enhanced features are compressed into a single-channel heatmap through 1×1 convolution, and the predicted heatmap is output after Sigmoid activation. The real heatmap is constructed into a two-dimensional Gaussian distribution based on the target center position. L1 Loss is used to calculate the difference between the predicted heatmap and the real heatmap to complete the center prediction task. A total loss function is constructed, which is a weighted sum of segmentation loss, center prediction loss, classification loss, and bounding box regression loss. The loss ratio of each task is balanced by the weight coefficients. The pixel attribution of overlapping targets is clarified by the segmentation task, and the false detection results of the background area are filtered by the center prediction task. The three tasks are coordinated to optimize the detection accuracy.

8. The image instance detection method according to claim 1, characterized in that, The method employs a dual-index non-maximum suppression (NMS) approach, utilizing both the cross-union ratio (CUIR) and the mask's Dessian coefficient, to filter the detection results. Based on the coordinate mapping relationship, the detection boxes and masks are restored to their original image dimensions, and the instance detection results are output, including: All detection results from multiple templates and multiple tasks are sorted in descending order of category confidence, and a set number of detection results with the highest confidence are retained as the detection results to be screened. Calculate the cross-union ratio (CUI) and mask Dess coefficient of the bounding boxes between the detection results to be screened. When the CUI of any two detection results to be screened reaches the set CUI and the mask Dess coefficient reaches the set coefficient, it is determined to be a duplicate prediction, and the detection result with higher confidence among the two detection results to be screened is retained. In the preprocessing stage of the query image, the coordinate mapping relationship between the original image and the standardized image is recorded. After the detection results are filtered, the detection box coordinates and mask pixel coordinates on the standardized image are reverse-converted according to the coordinate mapping relationship to restore the original image size. The output includes an instance identifier, category information, a precise bounding box, and a pixel-level mask. The spatial location of the detection result is consistent with that of the original image.

9. An image instance detection device, characterized in that, include: The preprocessing module is used to obtain multi-view templates of the target instance, standardize the multi-view templates and extract feature blocks through 3×3 ROI Pooling, generate template embedding vectors after viewpoint perturbation enhancement and store them offline. The feature extraction module is used to extract multi-scale features from the query image. It dynamically adjusts the feature extraction parameters through template feature branches to optimize the extraction of query image features. The feature fusion module projects the query image features and template embedding vectors into the same comparison space, generates an initial score map based on spatial correlation, and obtains a dynamic similarity map by fusing instance dimension normalization and spatial dimension normalization. Based on the dynamic similarity map as weights, the template features are aggregated by kernel flipping weighting, and the template features and query image features are fused through a learnable fusion structure to output enhanced features. The detection module is used to perform collaborative detection based on the enhanced features through three tasks: bounding box prediction, segmentation, and center prediction, and to optimize the detection accuracy using a weighted multi-task loss function. The post-processing module is used to filter the detection results using the cross-union ratio and mask Dessian coefficient as dual indicators of non-maximum suppression. Based on the coordinate mapping relationship, the detection box and mask are restored to the original image size, and the instance detection results are output.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that can be loaded by a processor and executed as described in any one of claims 1 to 8.

Images