Image target detection method and system based on double-input double-branch feature fusion

Abstract

This invention belongs to the field of target detection technology, specifically disclosing an image target detection method and system based on dual-input dual-branch feature fusion. By inputting the image to be identified and a reference template image into a shared backbone network, the feature maps to be identified and the template feature maps are extracted. Based on this, a dual-branch system of identification and reference branches is constructed. The identification branch enables refined screening and localization of candidate targets, while the reference branch fully mines the discriminative information of the reference template through differentiable soft pruning and a shared encoder, thereby improving the accuracy of similarity measurement between template features and the features of the target to be detected. Simultaneously, during the model training phase, a composite loss function is constructed and backpropagated to optimize network weights. The detection loss and similarity matching loss are weighted and jointly trained, enabling the model to simultaneously improve the feature consistency between candidate targets and the reference template while optimizing target location and category accuracy, thus enhancing the synergy between detection and matching.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to image target technology, and specifically discloses an image target detection method and system based on dual-input dual-branch feature fusion. Background Technology

[0002] Computer vision-based target detection is widely used in scenarios such as security monitoring and autonomous driving, where there is a need to detect rare or similar targets. Existing methods mostly achieve target recognition by extracting features from a single image to be detected. However, when faced with scarce samples and subtle differences between classes, their detection accuracy cannot meet the needs of practical applications.

[0003] To address the aforementioned issues, existing technologies introduce reference templates for auxiliary comparison. For example, Chinese invention patent publication number CN118135182A discloses a method, apparatus, and computing device cluster for target detection. Based on the template, a first region is determined by matching in the input image. Then, multiple first candidate regions are generated based on the first region. Finally, the target detection model is used to verify the candidate regions, and the target region and classification label are output.

[0004] The above scheme connects template matching with the detection model, but the template matching results are only used for coarse localization, and the fine-grained feature alignment and similarity measurement capabilities are insufficient. At the same time, the generation of candidate regions depends on Gaussian sampling of the features of the sample image data, lacks effective judgment of the texture of the candidate target itself, and is prone to introducing invalid regions.

[0005] For example, Chinese invention patent publication number CN119850920A discloses a target detection method, device, equipment and medium based on a fully convolutional Siamese network. The method extracts feature maps from the image to be detected and the template image through a convolutional layer with shared weights, then converts the feature maps into a pulse sequence using a pulse frequency algorithm, finally calculates the similarity of the pulse sequence to generate a response map, and determines the target position based on the offset of the response map.

[0006] The above scheme uses a twin network to achieve synchronous extraction of dual input features, but the feature encoding method it uses is relatively simple, resulting in insufficient feature representation between the template and the target to be detected, and the similarity matching is easily affected by background noise.

[0007] In summary, existing technologies do not make sufficient use of reference templates and lack fine extraction of features of the target to be detected, resulting in inaccurate similarity measurement between template features and target features.

[0008] Furthermore, target classification and template similarity matching are often performed in separate steps or simply in series, making it difficult to train detection loss and matching loss together, and thus the model cannot simultaneously improve detection accuracy and feature discrimination ability. Summary of the Invention

[0009] To address or at least partially address the aforementioned technical problems, this invention provides an image target detection method and system based on dual-input dual-branch feature fusion. Through dual-branch collaboration, refined feature extraction, and joint optimization, the detection accuracy for rare targets, similar targets, and small sample scenarios is significantly improved.

[0010] The objective of this invention can be achieved through the following technical solutions: In a first aspect, this invention proposes an image target detection method based on dual-input dual-branch feature fusion, comprising: acquiring the image to be identified and a reference template image, and performing size and pixel normalization preprocessing.

[0011] The preprocessed dual-channel images are input into a shared backbone network to extract the feature map to be identified and the template feature map.

[0012] Construct a dual-branch system, including: an identification branch: predicting the initial bounding box based on the feature map to be identified, filtering by the intersection-union ratio of the central observation region and the variance of the activation value, and outputting the candidate target bounding box and the category confidence.

[0013] Reference branch: Global pooling and dimension mapping are performed on the template feature map to obtain the template feature sequence; based on the candidate target bounding box, differentiable soft cropping is used to extract the candidate target feature block on the feature map to be identified, and the candidate target feature sequence is obtained by aligning with the padding mask. The two sequences are input into the shared encoder to output the template feature vector and the candidate target feature vector.

[0014] Calculate the cosine similarity between two feature vectors.

[0015] During the training phase, a composite loss function is constructed and backpropagation is used to optimize the network weights. During the inference phase, the category confidence and cosine similarity are fused to output the detection results.

[0016] Secondly, this invention proposes an image target detection system based on dual-input dual-branch feature fusion, comprising: an input preprocessing module: acquiring the image to be identified and a reference template image, and performing size and pixel normalization.

[0017] Feature extraction module: Uses a shared backbone network to extract the feature map to be identified and the template feature map.

[0018] Branch identification module: Predicts initial bounding boxes based on the feature map to be identified, filters them by the intersection-union ratio of the central observation region and the variance of the activation value, and outputs candidate target bounding boxes and class confidence scores.

[0019] The reference branch module includes a differentiable soft cropping unit, a padding mask unit, and a shared encoder. The soft cropping unit extracts candidate target feature blocks on the feature map using differentiable soft cropping based on the candidate target bounding box. The padding mask unit aligns the candidate target feature blocks to a uniform length through a padding mask operation. The shared encoder encodes the template sequence and candidate sequence in a unified manner, outputting the template feature vector and the candidate target feature vector.

[0020] Similarity calculation module: Calculates the cosine similarity between two feature vectors.

[0021] Loss optimization module: Constructs a composite loss function during the training phase and updates the network weights through backpropagation.

[0022] Results output module: During the inference phase, the detection results are output by fusing category confidence and cosine similarity.

[0023] Combining all the above technical solutions, the positive effects of this invention are as follows: 1. This invention inputs the image to be identified and the reference template image into a shared backbone network, extracts the feature map to be identified and the template feature map, and on this basis, constructs a dual branch of recognition branch and reference branch. The recognition branch enables the refined screening and localization of candidate targets, and the reference branch fully mines the discriminative information of the reference template through differentiable soft pruning and shared encoder, thereby improving the similarity measurement accuracy between template features and the features of the target to be detected, and ultimately enhancing the detection accuracy in rare target and similar target scenarios.

[0024] 2. This invention constructs a composite loss function during the model training phase and optimizes the network weights through backpropagation. It performs weighted joint training of detection loss and similarity matching loss, so that while optimizing the accuracy of target location and category, the model can simultaneously improve the feature consistency between candidate targets and reference templates, thereby enhancing the synergy between detection and matching. Attached Figure Description

[0025] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the method steps of Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart illustrating the image target detection implementation process in this invention.

[0028] Figure 3 This is a system module connection diagram of Embodiment 2 of the present invention;

[0029] Figure 4 This is a unit composition diagram of the reference branch module in this invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] In computer vision-based target detection, a target image is usually given, and the target with a specific appearance is required to be detected. If there is only one target in the target image that matches the feature, it is easy to detect. However, if there are many targets with similar appearances, or if there are targets that have never been seen before, it is easy to make false detections or misses. This is the requirement for detecting rare targets and similar targets.

[0032] Under such detection requirements, existing methods rely solely on feature extraction from a single image to be identified to achieve target recognition, which cannot effectively distinguish between visually similar target categories or those not present in the training samples.

[0033] To address this issue, for rare or similar targets, reference template images, such as example images of the target, can be provided. These templates are then used to perform region-by-region feature comparison and similarity measurement on the image to be identified. Even if the target's appearance is rare and highly similar to other targets, matching can still be achieved through the template's baseline features, thereby improving detection accuracy.

[0034] The specific implementation process of this invention is as follows:

[0035] Example 1.

[0036] See Figure 1 As shown, the present invention proposes an image target detection method based on dual-input dual-branch feature fusion, including: S1, acquiring the image to be identified and the reference template image, and performing size and pixel normalization preprocessing.

[0037] After acquiring the image to be recognized and the reference template image, their image sizes and pixel value ranges often differ because they may come from different acquisition devices. Directly inputting these into the subsequent network can lead to spatial scale mismatch during feature extraction, affecting detection accuracy and model convergence. Therefore, it is necessary to perform uniform size and pixel normalization preprocessing on both images.

[0038] In one specific embodiment, the size and pixel normalization preprocessing includes the following: scaling the image to be identified and the reference template image to a uniform spatial resolution. This operation eliminates the size difference between the two images, making them have the same height and width, for example, 640×640 pixels, thereby ensuring that the feature maps output by the subsequent shared backbone network have consistent spatial dimensions.

[0039] The pixel values ​​of the scaled image are linearly mapped from an integer range, such as 0 to 255, to a normalized floating-point range, such as [0, 1]. This operation can eliminate the difference in pixel value dimensions caused by contrast between different images, thus stabilizing the numerical range of the model input.

[0040] S2. Input the preprocessed dual-channel image into the shared backbone network to extract the feature map to be identified and the template feature map.

[0041] Although the preprocessed images have unified size and pixel range, they are essentially still pixel matrices and cannot be directly matched with the reference template and the image to be identified. They need to be converted into feature maps that computers can understand. To this end, this invention introduces a shared backbone network to map the two images to the same feature space and ensures that the feature extraction methods are completely consistent, thereby guaranteeing the effectiveness of subsequent similarity measurements.

[0042] As a preferred embodiment of the present invention, this step is specifically implemented as follows: S21, a deep convolutional neural network pre-trained on a general image dataset is used as the backbone network. The image to be identified and the reference template image share the network and the network parameters are completely shared, ensuring that the features output by the image to be identified and the reference template image are in the same distribution space.

[0043] S22. The shared backbone network outputs the feature map to be identified and the template feature map. Each feature map is a three-dimensional tensor, containing height, width, and channel dimensions. The height and width dimensions correspond to the spatial size of the feature map, preserving the positional contour information of the target; the channel dimension corresponds to the semantic abstraction level of the feature. The backbone network uses convolution and pooling operations to reduce the height and width of the feature map relative to the input image by a preset multiple, such as 32 times, thereby expanding the receptive field of each feature point and enabling it to contain a wider range of contextual information; at the same time, the number of channels is increased by a preset multiple to enhance the expressive power of the features.

[0044] S23. Each spatial location on the feature map is called a feature point, and each feature point corresponds to a C-dimensional feature vector, where C is the number of channels. The C-dimensional feature vector condenses the high-level semantic information of the local image patch at the feature point location. To quantify the information richness of each feature point, this invention defines the L2 norm of the feature vector of each feature point as the activation value of that feature point. The larger the activation value, the stronger the feature response at that location, which usually corresponds to a texture area.

[0045] In particular, by introducing activation values, the variance of activation values ​​within the projected region can be calculated during subsequent candidate target screening to assess whether there is sufficient texture variation in the region, thereby eliminating background regions lacking effective information and improving the quality of candidate targets.

[0046] S3. Construct a dual branch, including: a) Identification branch: predict the initial bounding box based on the feature map to be identified, filter by the intersection-union ratio of the central observation region and the variance of the activation value, and output the candidate target bounding box and the category confidence.

[0047] b) Reference branch: Global pooling and dimension mapping are performed on the template feature map to obtain the template feature sequence; based on the candidate target bounding box, differentiable soft clipping is used to extract the candidate target feature block on the feature map to be identified, and the candidate target feature sequence is obtained by aligning with the padding mask. The two sequences are input into the shared encoder to output the template feature vector and the candidate target feature vector.

[0048] After converting the dual-path images into feature maps, features can be extracted from these feature maps for similarity matching. However, since they carry different information: the feature map to be identified contains spatial and semantic information, it is necessary to locate the possible locations of the target as candidates; while the template feature map contains category-related global semantic information, so it is necessary to extract discriminative category features. Therefore, a separate processing branch is designed for each path of images.

[0049] Among them, the identification branch is the branch that performs preliminary target localization and screening on the feature map to be identified.

[0050] Since each feature point on the feature map to be identified corresponds to a local region of the original image, and the target may appear at any location, a fully convolutional detection head is required to perform dense predictions at all locations. Specifically, the identification branch is used for the following operations: a-1: Input the feature map to be identified into the fully convolutional detection head, where the detection head is a set of convolutional layers whose function is to map each feature point on the feature map to prediction parameters. Specifically, the detection head outputs a bounding box parameter set and a class probability vector in parallel for each feature point. The bounding box parameter set includes the bounding box center coordinates, width, and height. These parameters are used to determine the position and size of a candidate box in the original image coordinate system; the class probability vector is a vector with a length equal to the total number of classes, which is converted into the probability of each class through a flexible maximum function, and is used to determine the class of the target within the candidate box.

[0051] a-2: Initial bounding boxes are obtained by decoding the bounding box parameter set predicted for each feature point. Decoding involves converting the center coordinates of the bounding box to absolute coordinates in the original image coordinate system. This initial bounding box reflects the region where the feature point believes the target may exist. Simultaneously, the maximum value in the class probability vector is taken as the class confidence of the bounding box. In this operation, each initial bounding box is predicted by its corresponding feature point. The position of this feature point on the feature map to be recognized corresponds to the geometric center of the bounding box in the original image. This property ensures accurate mapping back to the feature map, forming the basis for subsequent projection and cropping.

[0052] a-3: Since the initial bounding boxes are predicted based on each feature point, the number of initial bounding boxes is enormous. In order to further filter out high-quality candidate boxes and avoid performing fine feature extraction and similarity calculation on all initial bounding boxes, considering that targets in natural images often appear in the central area of ​​the image and the edge areas are mostly background targets, the geometric center of the image is used as the defined rectangular area as the central observation area, which contains the most likely target area.

[0053] In one example, the size of the central observation area can be set to a fixed ratio of the image width and height, such as a 50% x 50% area in the center.

[0054] a-4: Calculate the intersection-union ratio (IU) between each initial bounding box and the central observation region. This quantifies the degree of overlap between the initial bounding box and the central observation region. The higher the IU, the more likely the initial bounding box is to be the target region. This process eliminates bounding boxes with IU below the admission threshold. The remaining bounding boxes are those candidate boxes located near the image center and with sufficient overlap with the central observation region. The admission threshold can be set to 0.6, meaning that bounding boxes with IU exceeding half of the central observation region are retained.

[0055] a-5: Although the retained bounding boxes are spatially close to the center, this does not necessarily mean they are the true targets. Therefore, the retained bounding boxes are then mapped to the coordinate system of the feature map to be identified according to the feature map downsampling ratio to obtain the projection area of ​​the bounding boxes on the feature map. This allows the bounding box positions in the original image space to be mapped to their spatial positions in the feature map, thus enabling the extraction of features at the feature map level. Then, the variance of the activation values ​​of all feature points in the projection area is calculated as the energy concentration. The energy concentration reflects the degree of fluctuation of the feature response in the projection area. If the target area has rich texture information, its activation value variance is usually large; conversely, if the area is the background, the variance will be small. Bounding boxes with energy concentration below the energy threshold are then removed to eliminate bounding boxes with poor texture and that are likely to be false detections. For example, the energy threshold can be taken as the median of the energy concentration of all retained bounding boxes.

[0056] a-6: The bounding boxes retained after the above double screening have high quality in terms of spatial location and texture validity. Therefore, the remaining bounding boxes are used as candidate targets, and the candidate target bounding boxes and class confidence scores are output.

[0057] The reference branch is used to perform discriminative feature extraction on the template feature map and to uniformly encode the candidate target features. Specifically, the reference branch is used for the following operations: b-1: The template feature map contains global semantic information of the reference target, but the spatial dimension is large and the features at each position are redundant. Therefore, global pooling is performed on the template feature map to compress the spatial dimension of each channel into a single value, resulting in a global feature vector of the same length as the number of channels, while retaining the class-related response. Then, a 1×1 convolutional layer is used to map the dimension of this vector to a preset uniform length, such as 256 dimensions, to form a template feature sequence, which is convenient for subsequent joint encoding with the candidate target feature sequence.

[0058] b-2: Since the bounding boxes of candidate targets usually cannot accurately fit the target contour, direct hard cropping will lose the contextual information near the boundary. This invention adopts differentiable soft cropping. Based on the bounding boxes of candidate targets, differentiable soft cropping is used to extract the feature blocks of candidate targets on the feature map to be identified. Differentiable soft cropping includes the following sub-steps: b-2-1: The bounding boxes of candidate targets are mapped to the coordinate system of the feature map to be identified according to the downsampling ratio of the feature map. The rectangular region of interest is defined by the projection area of ​​the bounding box, so that the position of the bounding box in the original image can accurately correspond to the spatial range on the feature map, thereby locating the region to be cropped at the feature map level.

[0059] b-2-2: Using the center of the mapped bounding box as the center of the Gaussian kernel, and setting the horizontal and vertical standard deviations of the width and height of the mapped bounding box respectively, a two-dimensional Gaussian mask with the same size as the region of interest is generated. The Gaussian function value at each position in the mask is used as the mask value. Since the mask has a bell-shaped distribution with a high center and low edges, it can give higher weights to the center position of the region of interest and gradually reduce the weights to the edge positions, thereby smoothly extracting the features of the target subject and suppressing background interference.

[0060] b-2-3: Using the mask value as weight, the C-dimensional feature vector of each pixel position in the region of interest is multiplied with the mask pixel by pixel to obtain a weighted feature block as a candidate target feature block. Through weighted fusion, the features near the boundary in the region of interest are weakened, while the central features are enhanced, making the extracted feature block more focused on the target itself.

[0061] b-3: Since the bounding box sizes of different candidate targets are different, the spatial dimensions of the clipped feature blocks are different, which cannot be directly input into the shared encoder. It is necessary to align the candidate target feature blocks through a padding mask to obtain the candidate target feature sequence. Specifically, the following sub-steps are included: b-3-1: Flatten the candidate target feature blocks obtained by soft clipping into a one-dimensional feature sequence, retaining the channel dimension. After flattening, each feature block becomes a sequence of length (height × width), and each position corresponds to a C-dimensional feature vector. Since the sequence length depends on the spatial size of the feature block, the sequence length after flattening is not fixed due to the different bounding box sizes of different candidate targets.

[0062] b-3-2; A unified target length is preset, for example, 256. For feature sequences whose flattened sequence length is less than the target length, zeros are padded to the end to the target length. For feature sequences whose length exceeds the target length, linear interpolation is used to compress them to the target length. Zero padding and interpolation compression are simple and differentiable operations that can force all candidate target feature sequences to be aligned to the same length, resulting in aligned candidate target feature sequences to meet the input requirements of the shared encoder.

[0063] b-3-3; While obtaining the aligned candidate target feature sequence, a binary mask of the same length as the target is generated simultaneously. The positions introduced by padding with zeros are assigned 0, and the remaining positions are assigned 1. This mask is used in the subsequent self-attention calculation of the shared encoder. By resetting the attention weight of the positions where the mask is 0 to negative infinity, the model ignores these invalid positions and only focuses on the original valid features, thereby ensuring the accuracy of the encoding result.

[0064] b-4: Following the aforementioned steps, we obtain the template feature sequence and the candidate target feature sequence. The template feature sequence contains global discrimination information of the reference target, while the candidate target feature sequence contains local region features of each candidate target. However, they originate from different feature extraction paths, lack contextual interaction within their sequences, and the candidate target feature sequence also contains invalid positions introduced by padding. To map the template and candidate targets to the same feature space, enhance the feature correlation within their respective sequences, and mask invalid information, both sequences need to be input into a shared encoder for unified encoding. Specifically, a shared encoder is used in this implementation.

[0065] The shared encoder employs a Transformer architecture, comprising multiple encoding layers. Each layer consists of a multi-head self-attention network and a feedforward network. The encoded output is then pooled to obtain a fixed-length feature vector, facilitating subsequent cosine similarity calculation. The specific steps are as follows: b-4-1: Input the template feature sequence and each candidate target feature sequence, along with their corresponding binary masks, into the encoder. In the multi-head self-attention calculation of each layer, the attention weights with a mask of 0 are reset to negative infinity or set to zero based on the mask, preventing them from participating in context aggregation. This avoids invalid features introduced by padding or compression interfering with the encoding of valid features.

[0066] b-4-2: The encoder output is an encoded sequence of the same length as the input sequence. In order to obtain a fixed-dimensional global feature representation, global average pooling is performed on the encoded output corresponding to the template sequence to obtain the template feature vector; global average pooling is performed on the encoded output corresponding to each candidate target sequence to obtain the feature vector of each candidate target.

[0067] S4. Calculate the cosine similarity between two feature vectors.

[0068] The candidate target feature vector output by the shared encoder is compared pairwise with the reference template feature vector. This invention uses cosine similarity to measure the feature consistency between the two, providing an auxiliary judgment basis for target recognition. In one example, the cosine similarity calculation formula is: .

[0069] in For the first Each candidate target feature vector For the first One reference template feature vector, Represents the L2 norm. express and The cosine similarity between the two feature vectors is 1 when the angle is 0°, indicating that the two feature vectors are most similar; when the angle is 180°, the cosine value is -1, indicating that the two feature vectors are completely opposite.

[0070] When multiple reference templates exist, the template feature vector that matches the candidate target category is selected as the positive sample, and the remaining template feature vectors and background region feature vectors are selected as the negative samples, which further improves the reliability of feature matching and avoids mismatches.

[0071] S5. During the training phase, a composite loss function is constructed and backpropagation is used to optimize the network weights. During the inference phase, the category confidence and cosine similarity are fused to output the detection results.

[0072] After calculating the cosine similarity between two feature vectors, this similarity is only used for output judgment during the inference stage and cannot be directly used in model training. Model training relies on a loss function, but a single detection loss cannot constrain feature similarity, resulting in insufficient ability of the model to distinguish between similar and rare targets. To address this, this invention constructs a composite loss function that weights and combines the detection loss and similarity matching loss. Through backpropagation, the backbone network, detection head, and shared encoder are simultaneously optimized, enabling the model to enhance the discriminative power of feature matching while maintaining basic detection accuracy.

[0073] To achieve the above joint optimization, we first need to assign positive and negative sample labels to the bounding boxes predicted for each feature point in the recognition branch, so as to calculate the classification loss and location loss in the detection loss.

[0074] Specifically, the intersection-union (IU) ratio of the predicted bounding box for each feature point with all ground truth bounding boxes is calculated. The ground truth bounding boxes are manually labeled target bounding boxes from the training data. If the IU ratio of a predicted bounding box with a ground truth box is greater than a positive sample threshold (e.g., 0.5), the predicted box is labeled as a positive sample, and its class label is the corresponding ground truth class. If the IU ratio of a predicted bounding box with all ground truth boxes is less than a negative sample threshold (e.g., 0.3), it is labeled as a negative sample, and its class label is background. Predicted bounding boxes with IU ratios between these thresholds are ignored during training. The classification loss only calculates the class prediction error for positive and negative samples.

[0075] For similarity matching loss, a feature library of positive and negative sample reference templates needs to be constructed.

[0076] Positive Sample Reference Template: For each candidate target, the feature vector of the reference template corresponding to its true category is taken as the positive sample.

[0077] Negative sample reference template: This includes feature vectors from reference templates of other categories, as well as feature vectors from the background region. The background region feature vector is obtained by randomly sampling regions without targets from the reference template image, and then performing the same feature extraction and encoding process (global pooling, dimension mapping, shared encoder).

[0078] When calculating the loss each time, a certain number of positive and negative sample feature vectors are randomly sampled for each candidate target to calculate the cosine similarity matching loss.

[0079] The composite loss function is composed of a weighted sum of detection loss and similarity matching loss, as follows: (1) Detection loss .

[0080] in, Represents classification loss. ,in Predict the candidate object bounding box as the true class The probability, As a class balance factor, For focusing parameters.

[0081] The detection loss reduces the weight of easily classified samples, making the model focus more on difficult-to-classify samples, thus alleviating the class imbalance problem.

[0082] Indicates position loss. ,in Indexing the bounding boxes of candidate targets, The total number of candidate target bounding boxes. To predict bounding box parameters, These are the parameters of the actual bounding box; .

[0083] Location loss is insensitive to outliers and can effectively avoid gradient explosion.

[0084] (2) The similarity matching loss adopts the cosine loss function to optimize the feature similarity matching effect, constrain the feature distance between the candidate target and the positive sample reference template to be as close as possible and the feature distance between the candidate target and the negative sample reference template to be as far as possible, thereby enhancing the feature discrimination.

[0085] The specific expression is as follows In the formula The cosine similarity between the candidate target and the positive sample reference template. The cosine similarity between the candidate target and the negative sample reference template. The similarity threshold is preferably 0.7.

[0086] (3) Total loss , The weighting coefficient ranges from 0.1 to 0.5. A preferred embodiment is... .

[0087] The total loss calculated above reflects the model's accuracy in target classification and its discriminative ability in feature matching; the smaller the value, the better the model's performance.

[0088] The implementation strategy for optimizing network weights through backpropagation is as follows: The network weights of the backbone network, detector head, and encoder are updated using the backpropagation algorithm based on the total loss; the optimizer is AdamW, with an initial learning rate of 1e-4 and a weight decay coefficient of 1e-5 to avoid overfitting; a gradient clipping threshold of 1.0 is set to prevent gradient explosion; training is iterated until the model's detection accuracy on the validation set shows no improvement for 10 consecutive rounds, at which point training is terminated using an early stopping strategy; the batch size is set to 16, the maximum number of iterations is 100 rounds, and an early stopping strategy is adopted: training is terminated when the model's detection accuracy on the validation set shows no improvement for 10 consecutive rounds.

[0089] After model training, the network weights have been fully optimized. At this point, the calculated cosine similarity and class confidence can be used for inference detection, and the final result can be output. Specifically, this includes the following: multiplying the class confidence of each candidate target by the cosine similarity to obtain a fusion score. This score comprehensively reflects the credibility of the candidate target in class determination and its feature consistency with the reference template.

[0090] Candidate targets with a fusion score greater than or equal to the effective score (e.g., 0.7) are selected as valid detection results, and the location bounding box, category label, and fusion score of the valid detection results are output.

[0091] For the entire implementation process of image object detection, please refer to [link / reference]. Figure 2 As shown.

[0092] Example 2.

[0093] See Figure 3 As shown, the present invention proposes an image target detection system based on dual-input dual-branch feature fusion, including: an input preprocessing module: acquiring the image to be identified and a reference template image, and performing size and pixel normalization.

[0094] Feature extraction module: Together with the input preprocessing module, it uses a shared backbone network to extract the feature map to be identified and the template feature map.

[0095] Recognition Branch Module: Connected to the feature extraction module, it predicts the initial bounding box based on the feature map to be recognized, filters it by the intersection-union ratio of the central observation region and the variance of the activation value, and outputs the candidate target bounding box and the category confidence.

[0096] Reference branch module: Connects to the feature extraction module; see [link / reference] Figure 4 As shown, it includes a differentiable soft cropping unit, a padding mask unit, and a shared encoder. The soft cropping unit extracts candidate target feature blocks on the feature map using differentiable soft cropping based on the candidate target bounding box. The padding mask unit aligns the candidate target feature blocks to a uniform length through a padding mask operation. The shared encoder encodes the template sequence and candidate sequence in a unified manner and outputs the template feature vector and the candidate target feature vector.

[0097] Similarity calculation module: Connected to the reference branch module, it calculates the cosine similarity between two feature vectors.

[0098] Loss optimization module: Connected to the reference branch module, it constructs a composite loss function during the training phase and backpropagates to update the network weights.

[0099] The results output module is connected to the similarity calculation module and the recognition branch module respectively. During the inference stage, it integrates the category confidence and cosine similarity to output the detection results.

[0100] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0101] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0103] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0104] Finally, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image target detection method based on dual-input dual-branch feature fusion, characterized in that, include: Obtain the image to be identified and the reference template image, and perform preprocessing for size and pixel normalization; The preprocessed dual-path images are input into a shared backbone network to extract the feature map to be identified and the template feature map. Build a two-branch structure, including: Identification Branch: Predicts the initial bounding box based on the feature map to be identified, filters it by the intersection-union ratio of the central observation region and the variance of the activation value, and outputs the candidate target bounding box and the category confidence. Reference branch: Perform global pooling and dimension mapping on the template feature map to obtain the template feature sequence; based on the candidate target bounding box, map the candidate target bounding box to the coordinate system of the feature map to be identified according to the feature map downsampling ratio, and define the rectangular region of interest by the projection area of ​​the bounding box; Using the center of the mapped bounding box as the center of the Gaussian kernel, and setting the horizontal and vertical standard deviations for the width and height of the mapped bounding box respectively, a two-dimensional Gaussian mask with the same size as the region of interest is generated, and the Gaussian function value at each position in the mask is used as the mask value. Using the mask value as weight, the C-dimensional feature vector of each pixel position in the region of interest is multiplied with the mask pixel by pixel to obtain a weighted feature block as a candidate target feature block; The candidate target feature sequence is obtained by aligning with the padding mask, and the two sequences are input into the shared encoder to output the template feature vector and the candidate target feature vector. Calculate the cosine similarity between two feature vectors; During the training phase, a composite loss function is constructed and backpropagation is used to optimize the network weights. During the inference phase, the category confidence and cosine similarity are fused to output the detection results.

2. The image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that: The size and pixel normalization preprocessing includes the following: Scaling the image to be identified and the reference template image to a uniform spatial resolution; The pixel values ​​of the scaled image are linearly mapped from the integer range to a normalized floating-point range.

3. The image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that: The extraction process for the feature map to be identified and the template feature map is as follows: A deep convolutional neural network pre-trained on a general image dataset is used as the backbone network. The image to be identified and the reference template image share the same network and the network parameters are completely shared. The shared backbone network outputs the feature map to be identified and the template feature map. Each feature map is a three-dimensional tensor, which includes height dimension, width dimension and channel dimension. Each spatial location on the feature map is called a feature point, and each feature point corresponds to a C-dimensional feature vector, where C is the number of channels; The L2 norm of the eigenvector of each feature point is defined as the activation value of the feature point.

4. The image target detection method based on dual-input dual-branch feature fusion as described in claim 3, characterized in that: The identification branch is used for the following operations: The feature map to be identified is input into the fully convolutional detection head, which predicts a bounding box parameter set and a class probability vector for each feature point. The bounding box parameter set includes the center coordinates, width and height of the bounding box, and the class probability vector is transformed into the probability of each class through the flexible maximum function. The initial bounding box is obtained by decoding the bounding box parameter set predicted by each feature point, and the maximum value in the class probability vector is taken as the class confidence of the bounding box. A rectangular region is defined by the geometric center of the image as the central observation area; Calculate the intersection-union ratio (IUR) between each initial bounding box and the central observation region, and remove bounding boxes with IUR below the admission threshold; The retained bounding boxes are mapped to the coordinate system of the feature map to be identified according to the feature map downsampling ratio to obtain the projection area of ​​the bounding boxes on the feature map. The variance of the activation values ​​of all feature points in the projection area is extracted as the energy concentration, and then bounding boxes with energy concentration lower than the energy threshold are removed. The remaining bounding boxes are used as candidate targets, and the candidate target bounding boxes and class confidence scores are output.

5. The image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that: The padding mask alignment includes: Flatten the candidate target feature block into a one-dimensional feature sequence, retaining the channel dimension; A unified target length is preset. For feature sequences whose flattened sequence length is less than the target length, zeros are padded to the end to reach the target length. For feature sequences whose flattened sequence length exceeds the target length, linear interpolation is used to compress them to the target length, resulting in aligned candidate target feature sequences. Synchronously generate a binary mask of the same length as the target, where positions introduced by padding with zeros are assigned 0, and the remaining positions are assigned 1.

6. The image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that: The template feature vector and the candidate target feature vector are shown in the following output process: The template feature sequence and each candidate target feature sequence, along with their respective masks, are input into the shared encoder. In the self-attention calculation, the attention weights at the positions where the mask is 0 are reset to zero. The shared encoder contains multiple coding layers, each consisting of a multi-head self-attention network and a feedforward network. The encoded output is then subjected to global average pooling to obtain the template feature vector and the feature vectors of each candidate target.

7. The image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that: The composite loss function is composed of a weighted sum of detection loss and similarity matching loss, as detailed below: Detection loss ; in, Represents classification loss. ,in Predict the candidate object bounding box as the true class The probability, For category balance factor, For focusing parameters; Indicates position loss. ,in Indexing the bounding boxes of candidate targets, The total number of candidate target bounding boxes. To predict bounding box parameters, These are the parameters of the actual bounding box; ; Similarity matching loss In the formula The cosine similarity between the candidate target and the positive sample reference template. The cosine similarity between the candidate target and the negative sample reference template. The similarity threshold; Total loss , These are the weighting coefficients.

8. The image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that: The fusion category confidence and cosine similarity output detection results include the following: The fusion score is obtained by multiplying the category confidence of each candidate target by its cosine similarity. Candidate targets with fusion scores greater than or equal to the valid scores are selected as valid detection results, and their location bounding boxes, category labels, and fusion scores are output.

9. An image target detection system based on dual-input dual-branch feature fusion, used in the image target detection method based on dual-input dual-branch feature fusion as described in claim 1, characterized in that, include: Input preprocessing module: acquires the image to be recognized and the reference template image, and performs size and pixel normalization; Feature extraction module: Uses a shared backbone network to extract the feature map to be identified and the template feature map; Branch identification module: predicts the initial bounding box based on the feature map to be identified, filters it by the intersection-union ratio of the central observation area and the variance of the activation value, and outputs the candidate target bounding box and the category confidence. Reference branch module: contains differentiable soft clipping units, padding mask units, and a shared encoder; The soft cropping unit extracts candidate target feature blocks on the feature map using differentiable soft cropping based on the candidate target bounding box; The padding mask unit aligns the candidate target feature blocks to a uniform length through a padding mask operation; the shared encoder encodes the template sequence and candidate sequence in a unified manner, outputting the template feature vector and the candidate target feature vector; Similarity calculation module: Calculates the cosine similarity between two feature vectors; Loss optimization module: Constructs a composite loss function during the training phase and updates the network weights through backpropagation; Results output module: During the inference phase, the detection results are output by fusing category confidence and cosine similarity.

Images