An unseen level RGB image target detection method using template matching paradigm

Abstract

The application provides a kind of unseen level RGB image target detection method using template matching paradigm, which first, obtains monocular query image and reference image database constructed in advance;Second, the multi-level features of the query image and the reference image are extracted by the feature extraction network sharing parameters, and the feature embedding is completed by double instance normalization and convolution transformation;Then, based on the scale matching of adaptive pooling and the context matching of mixed dilated convolution, a high-dimensional mixed correlation feature with multiple scales and multiple receptive fields is constructed;Finally, the feature fusion is realized by geometric distance screening and 3D convolution aggregation, and the two-dimensional center position and scale parameter of the target are jointly output by the scale prediction branch and the offset prediction branch.The application does not need to pre-train the target model, has strong real-time detection, high positioning and scale prediction accuracy, good robustness, and can be widely used in intelligent monitoring, robot vision, industrial detection and other various unseen level target detection scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for detecting unseen RGB image targets using a template matching paradigm. Background Technology

[0002] With the rapid development of computer vision technology, object detection, as one of the core tasks, plays a crucial role in many fields. Traditional object detection methods mainly target known object categories, meaning the network needs to be trained on a large number of samples for each object category before it can effectively detect that category. However, in real-world applications, there are many unseen objects—objects for which the network has not been trained or learned their features—making accurate detection difficult for traditional methods.

[0003] In existing technologies, some deep learning-based monocular vision-based methods for detecting missing objects employing template matching paradigms locate targets by calculating the feature similarity between the query image and pre-acquired reference images. However, these existing technologies have certain limitations in practical applications. First, the multi-scale processing mechanism suffers from significant computational redundancy. Due to the strategy of physically scaling the query image, the network needs to perform a complete forward propagation feature extraction for each preset scale. The computational cost increases linearly with the number of predefined scales, greatly increasing system latency and making it difficult to meet the requirements of real-time detection. Second, scale prediction lacks a global perspective. Existing scale prediction branches rely only on local convolutional operations. The network can only perceive the local texture details of objects, lacking a global contextual awareness of the object's proportion in the entire image. This results in inaccurate scale estimation when faced with objects that are similar in appearance but different in size, affecting detection accuracy. Finally, location regression lacks spatial sensitivity and the ability to capture long-distance dependencies. Existing offset prediction branches do not introduce effective attention mechanisms. Relying solely on stacked convolutional layers makes it difficult to accurately focus on the geometric center of an object under complex background clutter, which can easily lead to drift in the localization results. Furthermore, the fixed receptive field of the convolution is difficult to adapt to the large deformation of the object and the requirements of contextual association, which further reduces the accuracy and robustness of the detection. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for detecting unseen RGB image targets using a template matching paradigm to address the aforementioned technical problems.

[0005] A method for detecting unseen RGB image targets using a template matching paradigm includes the following steps:

[0006] S1, Obtain a monocular query image containing the target object and a pre-built reference image database;

[0007] S2, obtain at least one reference image of the target object from the reference image database, and process the monocular query image and the reference image through a feature extraction network with shared parameters to obtain the corresponding query features and reference features;

[0008] S3, Perform feature embedding processing on the query feature and the reference feature respectively to obtain embedded features; wherein, the feature embedding processing includes: instance normalization and convolution transformation;

[0009] S4, Based on the embedded features, construct correlation features between the query features and the reference features at multiple different receptive field scales;

[0010] S5, the correlation features obtained at different receptive field scales are fused, and based on the fused multi-scale correlation features, the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image are jointly predicted through at least two parallel prediction branches.

[0011] In one embodiment, the method further includes the following steps prior to step S1:

[0012] Collect a multi-view image sequence of an object against a textured background to obtain the original image;

[0013] The object binary mask corresponding to each frame of the original image is extracted using a foreground segmentation algorithm to obtain the first processed image sequence;

[0014] The first processed image sequence is processed using structure-of-motion reconstruction technology to calculate the sparse 3D point cloud and the camera parameters corresponding to each frame.

[0015] Perform coordinate system normalization on the camera parameters to construct a standardized coordinate system centered on the target object, and transform all original camera poses to the standardized coordinate system.

[0016] Based on the transformed pose, the original image is cropped at the center and scaled to a uniform resolution to obtain a normalized reference image block, which is then stored together with the corresponding pose and camera parameters to form a target object reference image database.

[0017] In one embodiment, the feature extraction network uses a pre-trained deep convolutional neural network as its backbone architecture and extracts multiple feature maps from the shallow, middle and deep layers of the backbone network, respectively; wherein, the shallow feature maps retain the geometric texture details of the target object, and the deep feature maps contain the semantic context information of the target object.

[0018] In one embodiment, step S3 includes:

[0019] The input original feature tensor is subjected to the first layer of instance normalization to obtain normalized features;

[0020] The normalized features are input into the convolutional layer to perform linear transformation of the feature space and compression and dimensionality reduction of the feature channel dimension;

[0021] After activation by a nonlinear activation function, the embedded features are obtained through a second layer of instance normalization to achieve statistical consistency.

[0022] In one embodiment, step S4 includes:

[0023] S401, based on adaptive pooling scale matching, performs adaptive average pooling on the normalized reference features. According to a preset set of size adjustment parameters, it dynamically compresses the spatial resolution of the reference features to a specific proportion of the original size. Using the scaled-down reference features as convolution kernels, it performs sliding window convolution on the query features to generate a pooled correlation response map.

[0024]

[0025] in, Indicates the location of the query feature. Pooling correlation response plot, Represents feature index , Indicates the height of the feature map of the reference image. This represents the reference feature after normalization. This represents the embedded query features. Indicates the first Size adjustment parameters corresponding to each feature;

[0026] S402, based on context matching using hybrid dilated convolution, maintains the size of the reference feature and defines a set of dilation rates. For each dilation rate in the set, an inner product operation is performed with the reference feature at the query feature map location to generate a hole-related response map.

[0027]

[0028] in, Indicates the location of the query feature. Correlation response map of voids, Represents feature index Represents the set of expansion rates , These represent the vertical and horizontal indices within the convolution kernel space, respectively. Indicates the first Size adjustment parameters corresponding to each feature Indicates the height of the feature map of the reference image. This represents the width of the feature map in the reference image. Indicates query characteristics, This represents the reference feature after normalization. Indicates the first One expansion rate;

[0029] S403 stacks all pooled correlation response maps and hollow correlation response maps generated at different feature levels along the channel dimension to form a high-dimensional hybrid correlation feature that integrates multi-scale and multi-receptive field matching information:

[0030]

[0031] in, This indicates the correlation map at the feature level. Feature Index Query feature location Response diagram at the location, Indicates the number of feature layers , Indicates at the feature level Feature Index Query feature location Pooling correlation response plot at the location, Indicates at the feature level Feature Index Query feature location Correlation response diagram of voids.

[0032] In one embodiment, fusing the correlation features obtained at different receptive field scales includes:

[0033] The geometric distance matrix is ​​calculated using the pose information of the reference image. The K-nearest neighbor algorithm is used to select the K nearest neighbor reference viewpoints on the geometric distance matrix for each reference viewpoint. Based on the selection results, a subset of relevant responses consistent with the pose of the current reference viewpoint is selected in the reference view dimension. Aggregation modeling is performed in the local pose neighborhood using 3D convolution. Finally, the maximum response in the reference view dimension is selected to generate the fused multi-scale correlation features.

[0034] In one embodiment, the prediction branches include a scale prediction branch and a shift prediction branch;

[0035] The scale prediction branch takes the fused multi-scale correlation features as input, performs global average pooling on the multi-scale correlation features to obtain the global receptive field, and then performs upsampling after convolution and activation to obtain global semantic features. The global semantic features are concatenated with the original local features in the channel dimension, and the logarithmic scale value is regressed through the output convolutional layer. Finally, the predicted scale is obtained through exponential transformation.

[0036] The offset prediction branch uses an adaptive weighting mechanism based on the maximum response difference to suppress noise and obtain fused features. The fused features are then subjected to 3D convolution and max pooling, and then predicted based on the adaptive weighting mechanism and coordinate attention to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image.

[0037] In one embodiment, the scale prediction branch takes the fused multi-scale correlation features as input, performs global average pooling on the multi-scale correlation features to obtain the global receptive field, performs upsampling after convolution and activation to obtain global semantic features, concatenates the global semantic features with the original local features in the channel dimension, regresses the logarithmic scale value through the output convolutional layer, and then obtains the prediction scale through exponential transformation, including:

[0038] Global semantic features are extracted using the following formula:

[0039]

[0040] in, Represents global semantic features. Indicates an upsampling operation. Represents a non-linear activation function. This represents the convolution operation. This indicates a global average pooling operation. Indicates multi-scale correlation characteristics;

[0041] The prediction scale is obtained based on the global semantic features and the original local features using the following formula:

[0042]

[0043]

[0044] in, Indicates the relative scale factor. Indicates the convolution regression scale. Indicates the output convolutional layer. Represents a non-linear activation function. Indicates a feature fusion convolutional layer. Indicates multi-scale correlation characteristics. Represents global semantic features.

[0045] In one embodiment, the offset prediction branch employs an adaptive weighting mechanism based on the maximum response difference to suppress noise and obtain fused features. These fused features are then subjected to 3D convolution and max pooling, and predicted using an adaptive weighting mechanism and coordinate attention to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image, including:

[0046] The fused features are calculated using the following formula, followed by 3D convolution and max pooling:

[0047]

[0048]

[0049] in, Indicates the characteristics after fusion. Represents the correlation response plot. Indicates adaptive weights, Represents the normalized activation function. This indicates the mean calculation operation along the feature channel dimension. Relationship response plot The maximum response value;

[0050] The input features are pooled along both the horizontal and vertical directions to obtain feature vectors; attention weights are generated through convolution; the input features are multiplied by the horizontal and vertical attention weights to obtain the output features weighted and enhanced by the coordinate attention mechanism.

[0051]

[0052] in, This represents the output features after coordinate attention enhancement. This represents the input features of the coordinate attention module. The representation is based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the horizontal direction. Indicates based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the vertical direction;

[0053] The enhanced features are output through two convolution heads to obtain the object center heatmap and pixel offset map, respectively, so as to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameter of the target object relative to the reference image.

[0054] In one embodiment, the method further includes: a network training step, wherein the total loss function used for training is a weighted sum of classification loss and regression loss;

[0055] The classification loss is calculated using the following formula:

[0056]

[0057] in, Represents classification loss. This represents the total number of pixels involved in the loss calculation. Indicates the weight of positive samples. This represents a binary mask generated based on the center of the real object. This represents the network's prediction score graph. Indicates the weight of negative samples;

[0058] The regression loss is calculated using the following formula:

[0059]

[0060] in, Indicates regression loss, A mask representing the effective area where an object exists. Indicates the forecast scale, Represents the true scale. Indicates the predicted offset. Indicates the actual offset. This indicates the prevention of local minima where the denominator is zero. This represents the square of the norm.

[0061] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: The present invention can overcome the defects of the existing unseen RGB image target detection method using the template matching paradigm, such as large computational redundancy, inaccurate scale prediction, and easy position drift, so as to achieve real-time and accurate detection of unseen targets, reduce computational costs, and improve detection robustness. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a method for detecting unseen RGB image targets using a template matching paradigm in one embodiment.

[0063] Figure 2 This is a schematic diagram of the overall model architecture of an unseen-level RGB image target detection method using the template matching paradigm in one embodiment.

[0064] Figure 3 This is a schematic diagram illustrating image feature changes in a feature extraction network in one embodiment;

[0065] Figure 4This is a schematic diagram of the architecture of a decoupled scale prediction head and coordinate offset head in one embodiment. Detailed Implementation

[0066] Before describing the specific embodiments of the present invention, the overall concept of the present invention will be explained as follows:

[0067] This invention is mainly about the development of target detection process. Currently, existing target detection methods for unseen RGB images using the template matching paradigm suffer from problems such as large computational redundancy, inaccurate scale prediction, and easy drift in position localization.

[0068] The inventors, through analysis, discovered that the main reasons for the above problems are computational redundancy caused by physical scaling in multi-scale processing, lack of a global perspective in scale prediction, and lack of an effective attention mechanism in position regression. Therefore, the core improvement ideas of this invention are: 1. Employing adaptive pooling and hybrid dilated convolution to achieve multi-scale receptive field capture in a single forward propagation, thus solving computational redundancy; 2. Designing a scale prediction branch that integrates global and local features to improve scale prediction accuracy; 3. Introducing an adaptive weighting mechanism and a coordinate attention mechanism to improve the accuracy and robustness of position localization; 4. Optimizing feature embedding processing to improve feature matching accuracy. Based on the above ideas, this invention proposes a complete unseen-level RGB image target detection method, which can effectively solve the shortcomings of existing technologies.

[0069] After introducing the overall concept of the present invention, in order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0070] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this specification should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0071] For ease of understanding, the terms used in the embodiments of this invention are explained below:

[0072] Unseen targets: These are targets for which the network has not been trained beforehand or learned the features of such objects, and can be detected using only a reference image database.

[0073] Template matching paradigm: refers to a detection paradigm that achieves target localization and parameter prediction by calculating the feature similarity between the query image and a pre-acquired reference image (template).

[0074] Feature embedding refers to the process of normalizing and compressing the extracted original features to obtain a feature representation with statistical consistency and low redundancy.

[0075] Receptive field: In a convolutional neural network, the size of the region in the input image corresponding to a pixel in the output feature map of a certain layer is considered. The larger the receptive field, the richer the contextual information it can capture.

[0076] Hybrid dilated convolution: refers to a convolution method that uses multiple convolution kernels with different dilation rates to perform convolution operations, thereby capturing contextual information at multiple receptive field scales.

[0077] Coordinate attention: an attention mechanism that can capture long-distance spatial directional relationships and improve the network's sensitivity to target location. It achieves precise focusing on the target location by extracting features along the horizontal and vertical directions and generating attention weights.

[0078] Instance normalization: A feature normalization method that independently calculates the mean and variance of a single image feature map in the spatial dimension to improve the statistical consistency of features. It is suitable for template matching scenarios with single or small samples.

[0079] In one embodiment, such as Figure 1 As shown, a method for detecting unseen RGB image targets using a template matching paradigm is provided, such as... Figure 2 As shown, a general model architecture diagram of an unseen-level RGB image target detection method using a template matching paradigm is presented, including the following steps:

[0080] Step S1: Obtain a monocular query image containing the target object and a pre-built reference image database.

[0081] Specifically, the monocular query image is an RGB image containing the unseen target object to be detected. It can be acquired by image acquisition devices such as monocular cameras, surveillance cameras, and mobile phone cameras. During the acquisition process, it is necessary to ensure that the image is clear and avoid problems such as blurry images, severe occlusion, excessively strong or weak lighting affecting the subsequent feature extraction and detection accuracy. If the acquired image has problems such as noise or blur, preprocessing operations such as noise reduction and sharpening can be performed in advance to ensure that the image quality meets the detection requirements.

[0082] Based on this, the steps preceding step S1 also include:

[0083] Collect a multi-view image sequence of an object against a textured background to obtain the original image;

[0084] The object binary mask corresponding to each frame of the original image is extracted using a foreground segmentation algorithm to obtain the first processed image sequence;

[0085] The first processed image sequence is processed using structure-of-motion reconstruction technology to calculate the sparse 3D point cloud and the camera parameters corresponding to each frame.

[0086] Perform coordinate system normalization on the camera parameters to construct a standardized coordinate system centered on the target object, and transform all original camera poses to the standardized coordinate system.

[0087] Based on the transformed pose, the original image is cropped at the center and scaled to a uniform resolution to obtain a normalized reference image block, which is then stored together with the corresponding pose and camera parameters to form a target object reference image database.

[0088] Specifically, the reference image database is a pre-constructed database containing multi-view reference images of the same category as the target object to be detected, along with related parameters. Its construction process includes the following sub-steps:

[0089] S101, Collect a multi-view image sequence of the object against a textured background to obtain the original image; during the collection process, the target object needs to be placed in a background with obvious texture to avoid the difficulty of feature extraction caused by a solid color background. At the same time, multiple frames of images are collected from different angles and distances to form a multi-view image sequence, ensuring that the reference image can fully cover the geometric features and texture information of the target object, and provide sufficient appearance prior for subsequent feature matching and scale prediction.

[0090] S102, the object binary mask corresponding to each frame of the original image is extracted using a foreground segmentation algorithm to obtain the first processed image sequence. The foreground segmentation algorithm can adopt the semantic segmentation algorithm in the existing technology (such as U-Net, MaskR-CNN, etc.). By extracting the object binary mask, the environmental background interference in the original image can be removed, and only the region of the target object is retained, providing accurate object region information for subsequent 3D point cloud calculation and image normalization.

[0091] S103, the first processed image sequence is processed using the structure-reconstruction-motion technique to calculate the sparse 3D point cloud and the camera parameters corresponding to each frame of the image. The structure-reconstruction-motion technique can recover the sparse 3D point cloud of the target object from the 2D image sequence by performing feature point extraction, feature matching, and geometric constraint solving on the multi-view image sequence. At the same time, it can calculate the camera intrinsic parameters (such as focal length, pixel size, etc.) and extrinsic parameters (such as camera pose, shooting angle, etc.) corresponding to each frame of the image, providing a basis for subsequent coordinate system normalization and image preprocessing.

[0092] S104: Perform coordinate system normalization on the camera parameters, construct a standardized coordinate system centered on the target object, and transform all original camera poses to the standardized coordinate system. Specifically, use a foreground mask to filter out the object point cloud in the sparse 3D point cloud, calculate the geometric center and scale factor of the object point cloud, and construct a standardized coordinate system with the geometric center of the object as the origin. Then, based on the calculated scale factor and camera parameters, uniformly transform all original camera poses to this standardized coordinate system to ensure that reference images from different perspectives have a unified coordinate reference.

[0093] S105: Based on the transformed pose, the original image is center-cropped and scaled to a uniform resolution to obtain normalized reference image blocks. These normalized reference image blocks are then stored along with the corresponding pose and camera parameters to form a target object reference image database. The center-cropping operation removes redundant background areas from the image, retaining only the effective area centered on the target object. The uniform resolution scaling adjusts all reference image blocks to a preset fixed resolution, ensuring scale uniformity during subsequent feature extraction and avoiding feature mismatch issues caused by resolution differences. The final stored content includes normalized reference image blocks, corresponding camera intrinsic and extrinsic parameters, and pose parameters, providing comprehensive reference information for subsequent feature matching and detection.

[0094] Step S2: Obtain at least one reference image of the target object from the reference image database, and process the monocular query image and the reference image through a feature extraction network with shared parameters to obtain the corresponding query features and reference features.

[0095] Specifically, when selecting reference images from the reference image database, at least one reference image (usually 3-5 images) that is closest to the viewpoint of the target object in the query image can be selected based on the shooting angle and shooting distance of the query image. This ensures that the reference image and the target object in the query image have a certain viewpoint correlation, thereby improving the accuracy of feature matching. If the viewpoint of the query image is special, multiple reference images can be randomly selected to cover different viewpoints, ensuring that reference features with a high degree of matching with the query features can be found.

[0096] The network receives input data, including a monocular query image to be detected. and a set in a reference image database To ensure uniformity of feature scale, the reference image is preprocessed to a uniform fixed resolution before input.

[0097] Based on this, the feature extraction network uses a pre-trained deep convolutional neural network as its backbone architecture, and extracts multiple feature maps from the shallow, middle and deep layers of the backbone network respectively; wherein, the shallow feature map retains the geometric texture details of the target object, and the deep feature map contains the semantic context information of the target object.

[0098] Specifically, such as Figure 3 As shown, the feature extraction network uses a pre-trained deep convolutional neural network as its backbone architecture. Specifically, the VGG19 model can be used, as it has strong feature extraction capabilities and can effectively capture the texture details and semantic information of images. Alternatively, other pre-trained deep convolutional neural networks such as ResNet and MobileNet can also be used, as long as they can achieve multi-level feature extraction; this will not affect the implementation effect of the invention. To ensure consistency and efficiency in feature extraction, the query image and the reference image are processed using the same feature extraction network, and the network parameters are shared to avoid feature differences caused by different network parameters.

[0099] In the feature extraction process, instead of extracting only a single-level feature map, multiple feature maps are extracted from the shallow, mid-level, and deep layers of the backbone network. Shallow feature maps (such as the outputs of the first few layers of the backbone network) retain the geometric texture details of the target object, capturing local features such as edges and contours. Deep feature maps (such as the outputs of the later layers of the backbone network) contain the semantic context information of the target object, reflecting its overall features and category attributes. Mid-level feature maps fall between the shallow and deep layers, taking into account both texture details and semantic information. By extracting multi-level feature maps, a complete feature pyramid can be constructed, providing a foundation for subsequent multi-scale feature matching.

[0100] Let the query features of the l-th layer be... The reference feature is Where l∈{1,2,3} represents the feature layer number. Through the above feature extraction process, multi-level query features corresponding to the query image and multi-level reference features corresponding to the reference image can be obtained. The feature dimension is set according to the output of the backbone network.

[0101] Step S3: Perform feature embedding processing on the query feature and the reference feature respectively to obtain embedded features; wherein, the feature embedding processing includes: instance normalization and convolution transformation.

[0102] Specifically, the extracted query and reference features are normalized and their dimensions compressed to remove redundant information, resulting in statistically consistent embedded feature representations that improve the efficiency and accuracy of subsequent feature matching. Feature embedding processing shares weights, performing the same processing operations on both query and reference features.

[0103] Based on this, step S3 includes:

[0104] The input original feature tensor is subjected to the first layer of instance normalization to obtain normalized features;

[0105] The normalized features are input into the convolutional layer to perform linear transformation of the feature space and compression and dimensionality reduction of the feature channel dimension;

[0106] After activation by a nonlinear activation function, the embedded features are obtained through a second layer of instance normalization to achieve statistical consistency.

[0107] Specifically, step S3 includes the following steps:

[0108] S301 performs the first-level instance normalization on the input original feature tensor to obtain normalized features. Instance normalization (IN) differs from batch normalization in existing technologies. It independently calculates the mean and variance of the feature map of a single image in the spatial dimension and performs the operation of removing the mean and dividing by the standard deviation on the feature map. This can effectively reduce the impact of batch differences and improve the statistical consistency of features. It is especially suitable for template matching scenarios with single samples or small samples.

[0109] S302, the normalized features are input into the convolutional layer to perform linear transformation of the feature space and compression / dimensionality reduction of the feature channel dimensions. The convolutional layer uses a 3×3 convolutional kernel, which performs spatial linear transformation on the normalized features, captures the spatial correlation between features, and compresses / reduces the feature channel dimensions simultaneously. Specifically, in this embodiment, if the number of feature channels output by the backbone network (such as VGG19) is 512 dimensions, this convolutional layer can compress and map it to 128 dimensions, which removes redundant information in the features, reduces the complexity of subsequent calculations, and improves detection efficiency.

[0110] S303, after activation by a nonlinear activation function, undergoes a second layer of instance normalization to obtain statistically consistent embedded features. The nonlinear activation function can be the ReLU function, which can introduce nonlinear factors, enhance the network's expressive power, and alleviate the gradient vanishing problem. The second layer of instance normalization further improves the statistical consistency of the embedded features, ensuring that the query features and reference features are in the same feature space, laying the foundation for subsequent relevance calculations.

[0111] In summary, feature embedding operations The specific calculation process can be expressed as follows: ,in The input is the original feature tensor. Indicates instance normalization, This represents a 3×3 convolutional layer. This represents a non-linear activation function. Through the above processing, the embedded features corresponding to the query features can be obtained respectively. Embedded features corresponding to reference features ,in , These represent the height and width of the query feature map, respectively. , These represent the height and width of the reference feature map, respectively.

[0112] Step S4: Based on the embedded features, construct correlation features between the query features and the reference features at multiple different receptive field scales.

[0113] Specifically, by designing a multi-scale receptive field, fine-grained local correspondences and larger-scale contextual information are captured simultaneously during a single forward propagation, thus solving the problem of large computational redundancy in multi-scale processing in existing technologies.

[0114] Based on this, step S4 includes:

[0115] S401, based on adaptive pooling scale matching, performs adaptive average pooling on the normalized reference features. According to a preset set of size adjustment parameters, it dynamically compresses the spatial resolution of the reference features to a specific proportion of the original size. Using the scaled-down reference features as convolution kernels, it performs sliding window convolution on the query features to generate a pooled correlation response map.

[0116]

[0117] in, Indicates the location of the query feature. Pooling correlation response plot, Represents feature index , Indicates the height of the feature map of the reference image. This represents the reference feature after normalization. This represents the embedded query features. Indicates the first Size adjustment parameters corresponding to each feature;

[0118] S402, based on context matching using hybrid dilated convolution, maintains the size of the reference feature and defines a set of dilation rates. For each dilation rate in the set, an inner product operation is performed with the reference feature at the query feature map location to generate a hole-related response map.

[0119]

[0120] in, Indicates the location of the query feature. Correlation response map of voids, Represents feature index Represents the set of expansion rates , These represent the vertical and horizontal indices within the convolution kernel space, respectively. Indicates the first Size adjustment parameters corresponding to each feature Indicates the height of the feature map of the reference image. This represents the width of the feature map in the reference image. Indicates query characteristics, This represents the reference feature after normalization. Indicates the first One expansion rate;

[0121] S403 stacks all pooled correlation response maps and hollow correlation response maps generated at different feature levels along the channel dimension to form a high-dimensional hybrid correlation feature that integrates multi-scale and multi-receptive field matching information:

[0122]

[0123] in, This indicates the correlation map at the feature level. Feature Index Query feature location Response diagram at the location, Indicates the number of feature layers , Indicates at the feature level Feature Index Query feature location Pooling correlation response plot at the location, Indicates at the feature level Feature Index Query feature location Correlation response diagram of voids.

[0124] Specifically, such as Figure 4 As shown, step S4 includes the following steps:

[0125] S401, based on adaptive pooling scale matching, performs adaptive average pooling operation on the normalized reference features, dynamically compresses the spatial resolution of the reference features to a specific ratio of the original size according to the preset size adjustment parameter set, uses the reduced-size reference features as convolution kernels, performs sliding window convolution on the query features, and generates a pooling correlation response map.

[0126] The size adjustment parameter set consists of multiple size adjustment parameters. The system is composed of components used to dynamically adjust the compression ratio of reference features to adapt to target objects of different scales. The adaptive average pooling operation automatically adjusts the pooling window size according to the preset compression ratio, ensuring that the compressed reference features accurately reflect the core information of the original reference features. The reduced-size reference features are used as convolution kernels and subjected to sliding window convolution with the query features. Essentially, this calculates the similarity between the query features and the reference features at different locations, generating a pooling relevance response map. This response map reflects the local fine-grained correspondence between the query features and the reference features.

[0127] The formula for calculating the pooling correlation response plot is as follows:

[0128]

[0129] in, Indicates the location of the query feature. Pooling correlation response plot, Represents feature index and , Indicates the height of the feature map of the reference image. This represents the reference feature after normalization. This represents the embedded query features. Indicates the first The size adjustment parameters corresponding to each feature.

[0130] S402, based on context matching of hybrid dilated convolution, keeps the size of the reference feature unchanged, defines a set of dilation rates, and for each dilation rate in the set, performs an inner product operation with the reference feature at the query feature map location to generate a hole correlation response map;

[0131] Expansion rate set Among them, low expansion rate Focus on compact local features, moderate expansion rate Extending the receptive field to the level of object components, with a large expansion rate. This covers a larger global context and achieves contextual information capture at multiple receptive field scales through combinations of different expansion rates.

[0132] For sets Each inflation rate in the query feature map location Performing an inner product operation with the reference feature essentially expands the receptive field of the query feature through dilated convolution, capturing the long-distance contextual association between the query feature and the reference feature, and generating a dilated relevance response map. This response map can reflect the global contextual correspondence between the query feature and the reference feature, compensating for the limitations of local features in the pooling relevance response map.

[0133] The formula for calculating the void correlation response map is as follows:

[0134]

[0135] in, Indicates the location of the query feature. Correlation response map of voids, Represents the feature index and $ , Represents the set of expansion rates , These represent the vertical and horizontal indices within the convolution kernel space, respectively. Indicates the first Size adjustment parameters corresponding to each feature Indicates the height of the feature map of the reference image. This represents the width of the feature map in the reference image. Indicates query characteristics, This represents the reference feature after normalization. Indicates the first Expansion rate.

[0136] S403 stacks all pooled correlation response maps and hollow correlation response maps generated at different feature levels in the channel dimension to form a high-dimensional hybrid correlation feature that integrates multi-scale and multi-receptive field matching information.

[0137] Different feature levels ( , Pooled correlation response maps and hollow correlation response maps generated on the feature layers (representing the number of feature layers) correspond to matching information at different scales and receptive fields, respectively. Stacking these response maps along the channel dimension can fuse local fine-grained information and global contextual information to form high-dimensional hybrid correlation features, providing comprehensive matching information for subsequent feature fusion and prediction.

[0138] The expression for high-dimensional mixed correlation features is as follows:

[0139]

[0140] in, This indicates the correlation map at the feature level. Feature Index Query feature location Response diagram at the location, Indicates the number of feature layers , Indicates at the feature level Feature Index Query feature location Pooling correlation response plot at the location, Indicates at the feature level Feature Index Query feature location Correlation response diagram of voids.

[0141] Step S5: The correlation features obtained at different receptive field scales are fused. Based on the fused multi-scale correlation features, the two-dimensional center position of the target object in the monocular query image and the scale parameter of the target object relative to the reference image are jointly predicted through at least two parallel prediction branches.

[0142] Specifically, this step is divided into two parts: correlation feature fusion and dual-branch joint prediction. Through screening, fusion and branch prediction, the accuracy of scale prediction and location positioning is improved.

[0143] Based on this, the fusion of the correlation features obtained at different receptive field scales includes:

[0144] The geometric distance matrix is ​​calculated using the pose information of the reference image. The K-nearest neighbor algorithm is used to select the K nearest neighbor reference viewpoints on the geometric distance matrix for each reference viewpoint. Based on the selection results, a subset of relevant responses consistent with the pose of the current reference viewpoint is selected in the reference view dimension. Aggregation modeling is performed in the local pose neighborhood using 3D convolution. Finally, the maximum response in the reference view dimension is selected to generate the fused multi-scale correlation features.

[0145] Based on this, the prediction branches include scale prediction branches and migration prediction branches;

[0146] The scale prediction branch takes the fused multi-scale correlation features as input, performs global average pooling on the multi-scale correlation features to obtain the global receptive field, and then performs upsampling after convolution and activation to obtain global semantic features. The global semantic features are concatenated with the original local features in the channel dimension, and the logarithmic scale value is regressed through the output convolutional layer. Finally, the predicted scale is obtained through exponential transformation.

[0147] The offset prediction branch uses an adaptive weighting mechanism based on the maximum response difference to suppress noise and obtain fused features. The fused features are then subjected to 3D convolution and max pooling, and then predicted based on the adaptive weighting mechanism and coordinate attention to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image.

[0148] Based on this, the scale prediction branch inputs the fused multi-scale correlation features, performs global average pooling on the multi-scale correlation features to obtain the global receptive field, and then upsamples the global semantic features after convolution and activation to obtain global semantic features. The global semantic features are concatenated with the original local features in the channel dimension, and the logarithmic scale value is regressed through the output convolutional layer. Finally, the predicted scale is obtained through exponential transformation, including:

[0149] Global semantic features are extracted using the following formula:

[0150]

[0151] in, Represents global semantic features. Indicates an upsampling operation. Represents a non-linear activation function. This represents the convolution operation. This indicates a global average pooling operation. Indicates multi-scale correlation characteristics;

[0152] The prediction scale is obtained based on the global semantic features and the original local features using the following formula:

[0153]

[0154]

[0155] in, Indicates the relative scale factor. Indicates the convolution regression scale. Indicates the output convolutional layer. Represents a non-linear activation function. Indicates a feature fusion convolutional layer. Indicates multi-scale correlation characteristics. Represents global semantic features.

[0156] Based on this, the offset prediction branch uses an adaptive weighting mechanism based on the maximum response difference to suppress noise and obtain fused features. These fused features are then subjected to 3D convolution and max pooling, and predicted using the adaptive weighting mechanism and coordinate attention to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image, including:

[0157] The fused features are calculated using the following formula, followed by 3D convolution and max pooling:

[0158]

[0159]

[0160] in, Indicates the characteristics after fusion. Represents the correlation response plot. Indicates adaptive weights, Represents the normalized activation function. This indicates the mean calculation operation along the feature channel dimension. Relationship response plot The maximum response value;

[0161] The input features are pooled along both the horizontal and vertical directions to obtain feature vectors; attention weights are generated through convolution; the input features are multiplied by the horizontal and vertical attention weights to obtain the output features weighted and enhanced by the coordinate attention mechanism.

[0162]

[0163] in, This represents the output features after coordinate attention enhancement. This represents the input features of the coordinate attention module. The representation is based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the horizontal direction. Indicates based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the vertical direction;

[0164] The enhanced features are output through two convolution heads to obtain the object center heatmap and pixel offset map, respectively, so as to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameter of the target object relative to the reference image.

[0165] Specifically, step S5 includes the following steps:

[0166] S501, Relevance Feature Fusion: Calculate the geometric distance matrix using the pose information of the reference image, and use the K-nearest neighbor algorithm to select the K nearest neighboring reference viewpoints on the geometric distance matrix for each reference viewpoint. Based on the selection results, select a subset of relevant responses consistent with the pose of the current reference viewpoint in the reference view dimension, perform aggregation modeling through 3D convolution in the local pose neighborhood, and then generate fused multi-scale correlation features by selecting the maximum response in the reference view dimension.

[0167] The pose information of the reference image is contained in the reference image database. The geometric distance matrix is ​​used to represent the pose differences between different reference viewpoints. By calculating the geometric distance between reference viewpoints, the neighboring reference viewpoints whose poses are closest to the current reference viewpoint can be selected. The correlation responses corresponding to these neighboring reference viewpoints have higher reference value. The value of K in the K-nearest neighbor algorithm can be adjusted according to the actual application scenario. Usually, 3-5 neighboring reference viewpoints are selected to ensure the effectiveness of the correlation response while avoiding computational redundancy caused by too many neighbors.

[0168] Aggregation modeling via 3D convolution within the local pose neighborhood enables the fusion of relevance responses from multiple neighboring reference viewpoints, capturing correlation information across the reference viewpoint dimension. The selection of the maximum response across the reference viewpoint dimension filters out the most discriminative relevance response information, removing redundancy and noise interference, ultimately generating a fused multi-scale correlation feature tensor. .

[0169] S502, Dual-branch Joint Prediction: Two parallel prediction branches are set up, namely the scale prediction branch and the offset prediction branch. The two branches work together to jointly predict the two-dimensional center position and scale parameters of the target object.

[0170] (1) Scale prediction branch: Input the fused multi-scale correlation features, perform global average pooling on the multi-scale correlation features to obtain the global receptive field, and upsample after convolution and activation to obtain global semantic features. Concatenate the global semantic features with the original local features in the channel dimension, regress the logarithmic scale value through the output convolutional layer, and then obtain the prediction scale through exponential transformation.

[0171] Global average pooling (AvgPool) can obtain the global receptive field of multi-scale relevance features, capture the global contextual information of the target object in the whole image, and solve the problem of lack of global vision in scale prediction in existing technologies. Upsampling can restore the feature map after global average pooling to the same resolution as the original local features, which is convenient for subsequent feature stitching. Convolution and ReLU activation functions are used to further extract global semantic features and enhance the expressive power of features.

[0172] The formula for extracting global semantic features is as follows:

[0173]

[0174] in, Represents global semantic features. Indicates an upsampling operation. Represents a non-linear activation function. This represents the convolution operation. This indicates a global average pooling operation. This indicates multi-scale correlation characteristics.

[0175] The global semantic features are combined with the original local features (i.e., the fused multi-scale correlation features). By stitching along the channel dimension, both global contextual information and local texture details can be taken into account, improving the accuracy of scale prediction; output convolutional layer ( Used to regress logarithmic scale values Exponential transformation is used to convert logarithmic scale values ​​into actual prediction scales. The specific formula is as follows:

[0176]

[0177]

[0178] in, This represents the relative scale factor, which is the ratio between the projected size of the target object in the query image and its projected size in the reference image. Indicates the convolution regression scale. Indicates the output convolutional layer. Represents a non-linear activation function. Indicates a feature fusion convolutional layer. Indicates multi-scale correlation characteristics. Represents global semantic features. This indicates a channel splicing operation.

[0179] Based on the pinhole camera imaging model, the projected size of an object on the image plane is inversely proportional to its depth. Therefore, by combining known reference images, the depth can be captured. In addition to the camera focal length parameters, the depth of the target object in the query camera coordinate system can be further derived. The calculation formula is: This provides a foundation for subsequent three-dimensional positioning.

[0180] (2) Offset prediction branch: An adaptive weighting mechanism based on the maximum response difference is used to suppress noise and obtain fused features. The fused features are then predicted based on the adaptive weighting mechanism and coordinate attention after 3D convolution and max pooling to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image.

[0181] First, an adaptive weighting mechanism based on the maximum response difference is used to suppress noise interference in relevance features and improve feature purity. Adaptive weights The formula for calculating it is as follows:

[0182]

[0183] in, Indicates adaptive weights, This represents the normalized activation function, used to normalize the weights to a range of 0-1. This indicates the mean calculation operation along the feature channel dimension. Relationship response plot Maximum response value, This represents the correlation response map. By calculating the difference between the maximum response value and the response value at each location, and then performing mean and normalization processing, higher weights can be assigned to the effective regions (regions with high response values) in the correlation response map, and lower weights can be assigned to the noisy regions (regions with low response values), thereby suppressing noise.

[0184] Features after fusion The calculation formula is as follows:

[0185]

[0186] in, Indicates the characteristics after fusion. Represents the correlation response plot. This indicates adaptive weights.

[0187] Features after fusion After 3D convolution and max pooling, the data enters the coordinate attention module. The coordinate attention module can capture long-distance spatial orientation relationships, improving the network's spatial sensitivity to the center position of objects and solving the problem of position regression drift in existing technologies. Its specific operation is as follows:

[0188] Input features along the horizontal and vertical directions respectively Pooling is performed to obtain the feature vector. (Horizontal direction) and (Vertical direction); Attention weights are generated through convolutional layers. (Horizontal direction) and (Vertical direction); Multiply the input features by the horizontal and vertical attention weights to obtain the output features after weighting and enhancement by the coordinate attention mechanism. The calculation formula is as follows:

[0189]

[0190] in, This represents the output features after coordinate attention enhancement. This represents the input features of the coordinate attention module. Indicates based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the horizontal direction. Indicates based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the vertical direction.

[0191] The enhanced features are output through two convolutional heads: an object center heatmap and a pixel offset map. The object center heatmap is used to predict the probability of the existence of the target object center, and the pixel offset map is used to predict the fine pixel offset of the target object center relative to the peak position of the heatmap. Combining the two, the two-dimensional center position of the target object in the monocular query image can be obtained. At the same time, the scale parameters output by the scale prediction branch are combined to complete the full detection of the target object.

[0192] In addition, it also includes: network training steps, where the total loss function used for training is a weighted sum of classification loss and regression loss;

[0193] The classification loss is calculated using the following formula:

[0194]

[0195] in, Represents classification loss. This represents the total number of pixels involved in the loss calculation. Indicates the weight of positive samples. This represents a binary mask generated based on the center of the real object. This represents the network's prediction score graph. Indicates the weight of negative samples;

[0196] The regression loss is calculated using the following formula:

[0197]

[0198] in, Indicates regression loss, A mask representing the effective area where an object exists. Indicates the forecast scale, Represents the true scale. Indicates the predicted offset. Indicates the actual offset. This indicates the prevention of local minima where the denominator is zero. This represents the square of the norm.

[0199] Specifically, the total loss function used in training is a weighted sum of classification loss and regression loss, which is used to supervise network learning and improve detection accuracy.

[0200] (1) Classification loss The heatmap is generated using a weighted binary cross-entropy loss method to supervise its generation and is used to optimize the prediction of the existence probability of the target object's center. The calculation formula is as follows:

[0201]

[0202] in, Represents classification loss. This represents the total number of pixels involved in the loss calculation. Indicates the weight of positive samples. This represents a binary mask generated based on the center of the real object (the area near the center of the object is 1, and the rest of the area is 0). This represents the network's prediction score graph (i.e., the probability graph of the existence of the object center). This indicates the negative sample weight. and This is used to balance the difference in the number of positive and negative samples, and to prevent the network training from being biased towards negative samples due to an excessive number of negative samples.

[0203] (2) Regression loss Calculated only within the effective region where the object exists, it is used to optimize the prediction accuracy of scale parameters and 2D center position, and includes scale loss and offset loss. The calculation formula is as follows:

[0204]

[0205] in, Indicates regression loss, A mask representing the effective area where the object exists (object area is 1, background area is 0). Indicates the forecast scale, Represents the true scale. This represents the predicted offset (i.e., the pixel offset of the target object's center relative to the peak position of the heatmap). Indicates the actual offset. This indicates the prevention of local minima where the denominator is zero. This represents the square of the L2 norm.

[0206] Total loss function The weighted sum of classification loss and regression loss is given by the following formula: ,in and These are the weight coefficients for classification loss and regression loss, respectively, which can be adjusted according to the actual training effect.

[0207] Compared with the prior art, the present invention has the following significant advantages and beneficial effects:

[0208] 1. This invention solves the computational redundancy problem of multi-scale processing in existing technologies: By using scale matching based on adaptive pooling and context matching based on hybrid dilated convolution, this invention can capture matching information of multiple scales and multiple receptive fields in a single forward propagation of the backbone network. This eliminates the need for multiple physical scaling and repeated feature extraction of the query image, significantly reducing computational costs, reducing system latency, and improving the real-time performance of detection.

[0209] 2. Improved accuracy of scale prediction: The scale prediction branch of this invention integrates global semantic features and local features, and obtains the global receptive field through global average pooling. It can capture the global contextual information of the target object in the whole image, avoiding the problem of inaccurate scale prediction caused by relying only on local features in the prior art. It is especially suitable for the detection of target objects that are similar in appearance but different in size.

[0210] 3. Improved accuracy and robustness of location positioning: The offset prediction branch of this invention introduces an adaptive weighting mechanism and a coordinate attention mechanism based on the maximum response difference. The adaptive weighting mechanism can effectively suppress noise interference, and the coordinate attention mechanism can capture long-distance spatial orientation relationships and accurately focus on the geometric center of the object, avoiding drift of positioning results and improving the accuracy and robustness of target positioning in complex backgrounds.

[0211] 4. Applicable to unseen target detection scenarios: This invention does not require pre-training of the target object. It can achieve accurate detection of unseen targets using only a pre-built reference image database, thus expanding the application scope of target detection and meeting the needs of unknown target detection in fields such as intelligent monitoring and robot vision.

[0212] 5. High feature matching accuracy: This invention improves the statistical consistency between query features and reference features through dual instance normalization in feature embedding processing, ensuring that they are in the same feature space; at the same time, the fusion of multi-scale and multi-receptive field relevance features further improves the accuracy of feature matching, providing a reliable foundation for subsequent prediction.

[0213] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0214] The technical solutions of this invention will be clearly and completely described below with reference to the embodiments thereof. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0215] Example 1

[0216] This method is applied to intelligent monitoring scenarios to perform real-time detection of unseen targets (such as unknown types of packages, foreign objects, etc.) in the monitoring screen. The specific implementation steps strictly follow the technical solution of this invention, and clearly define the specific parameters, equipment selection, and operational details of each step to ensure that the method is reproducible, as follows:

[0217] 1. Construct a reference image database:

[0218] The target object is placed on a flat background with rich high-frequency textures (such as AprilTag markers or random noise patterns). A handheld monocular camera is used to take multi-angle, high-density surround shots in the upper hemisphere space of the object, and a video sequence with a frequency of no less than 30fps is collected to obtain the original image.

[0219] Foreground segmentation is performed on the image sequence after frame extraction from the video using interactive segmentation models (such as SAM) or background subtraction methods to generate a binarized mask aligned with the pixels of the original image, thus removing background interference. Next, the image sequence is processed using COLMAP to extract SIFT features and establish geometric constraints between images. Sparse 3D point clouds of the scene and the camera intrinsic parameter matrix corresponding to each frame are then calculated through bundle adjustment and joint optimization. and original extrinsic pose .

[0220] The first image sequence was processed using Structure from Motion (SfM) technology to calculate the sparse 3D point cloud and the camera parameters corresponding to each frame. The SfM technology uses the COLMAP open-source framework, and the SIFT algorithm is used for feature point extraction to calculate the camera intrinsic parameters.

[0221] To eliminate scale arbitrariness, the system performs coordinate system normalization: First, the geometric center of the foreground point cloud is calculated, and the origin of the coordinate system is translated to this center; second, the 3D bounding box diameter of the point cloud is calculated, and all point cloud coordinates and camera translation vectors are divided by this diameter, so that the object is scaled to within a unit sphere; finally, all camera poses are transformed to this new normalized coordinate system to obtain the normalized pose. Based on the normalized pose and intrinsic parameters, the object region in each frame of the original image is projected to the image center, and square cropping and bilinear interpolation scaling are performed to generate a standard reference image patch with a uniform resolution of 120×120 pixels.

[0222] 2. Feature extraction:

[0223] From the reference image database, based on the shooting angle and shooting distance of the query image in the monitoring screen, select 3 reference images that are closest to the perspective of the package in the query image to ensure that the reference images and the package in the query image have a high degree of perspective correlation.

[0224] The feature extraction network uses a pre-trained VGG19 model as its backbone architecture. This model was pre-trained on the ImageNet dataset, and the pre-trained weights are retained without additional fine-tuning. The query image and the reference image are processed using the same VGG19 model, and the network parameters are shared to ensure consistency in feature extraction.

[0225] The network processes the input query image (resolution adjusted to a multiple of 120×120, such as 480×640) and the reference image (120×120). The feature extraction network uses the VGG19-BN model, with its parameters frozen to maintain stability. The system extracts the output features from Layer 0, Layer 1, and Layer 2 of the VGG19 model, respectively.

[0226] 3. Feature embedding processing:

[0227] The specific network parameter configuration in this embodiment is as follows: For the input feature tensor (512 channels), the embedding module first undergoes a first-layer instance normalization; then it enters a 1×1 convolutional layer to reduce the number of feature channels from 512 dimensions to 128 dimensions to reduce computational cost; next, it passes through a LeakyReLU activation function with a negative slope of 0.1; finally, it undergoes a second-layer instance normalization. The output of this module is a 128-channel embedding feature map, ensuring the efficiency of subsequent correlation calculations on a low-dimensional manifold.

[0228] 4. Construction of multi-scale correlation features:

[0229] The specific parameters for constructing the hybrid correlation volume are set as follows: for each level, d = 64 is used as the intermediate feature channel dimension. In "pooling-based matching", adaptive average pooling is performed on the reference feature map, downsampling its spatial resolution to 60×60 and 30×30 respectively (corresponding to 1 / 2 and 1 / 4 of the original 120×120).

[0230] In matching based on hybrid dilated convolution, the dilation rate set is set as follows: This means the system processes the query features using convolutional kernels with dilation rates of 1 (standard convolution), 2, and 3, respectively. For each dilation rate... The effective receptive field of the convolutional kernel is expanded, but the number of parameters remains unchanged. All calculated correlation maps are concatenated along the channel dimension to form a hybrid correlation feature with a total number of channels of 3 × (number of pooling layers + number of dilation layers). Then, a 1 × 1 convolutional layer is used to uniformly adjust the number of channels. This is to facilitate subsequent processing.

[0231] 5. Feature fusion and dual-branch joint prediction:

[0232] The specific implementation parameters for feature selection and prediction branches are as follows: In the scale prediction branch, firstly, based on the geometric distance matrix of the reference image, the KNN algorithm is used to select... The reference feature index of the nearest neighbors ( The input feature channel count is d=64. The global context module first compresses the feature space size to 1×1 using adaptive average pooling, then reduces the number of channels to 32 using 1×1 convolution, and restores it by upsampling after ReLU activation. During fusion, the global features (32 channels) and local features (64 channels) are concatenated to form 96 channels, and finally, the logarithmic scale value of 1 channel is regressed using 3×3 convolution. .

[0233] In the offset prediction branch, to suppress noise, an adaptive weighting mechanism calculates the difference between the maximum and mean of the feature volume in the channel dimension, and generates weights through Softmax. In the coordinate attention module, AdaptiveAvgPool2d((H, 1)) and AdaptiveAvgPool2d((1, W)) are used to generate feature descriptors in the horizontal and vertical directions, respectively. A shared convolutional layer reduces the number of channels to d / reduction (e.g., 64 / 16=4), and then a Sigmoid activation is used to generate attention weights. The final output layer contains two convolutional heads, outputting the scale and coordinates of the objects in the image, respectively.

[0234] In one exemplary experimental setup, the training process can be configured with parameters such as the following. Specifically, during training, we use the Adam optimizer, setting the total number of iterations to 300,000, the batch size to 8, and the initial learning rate to 10. -4 The learning rate was reduced by a factor of 5 after every 100,000 iterations. Training was performed on a synthetic dataset rendered using ShapNet and the coco dataset, and validation tests were conducted on the LINEMOD and GenMOP datasets.

[0235] 6. Network training:

[0236] The specific parameters of the loss function are set as follows:

[0237] For classification loss In generating a binary mask for ground truth labels At that time, the spatial distance threshold is set to 1.5 pixels. That is, only when the Euclidean distance between the grid point on the feature map and the center of the real object is less than 1.5, the point is marked as a positive sample (1), and the rest are negative samples (0).

[0238] For regression loss Similarly, an effective area mask is set. The distance threshold is 1.5 pixels. A scale balance coefficient is set when calculating the scale loss. The value is 1.0, and the specific calculation formula is as follows:

[0239]

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0241] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the invention as described above, which are not provided in the details for the sake of brevity.

[0242] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0243] While specific details have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive. Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description.

[0244] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the protection scope of this invention.

Claims

1. A method for detecting unseen RGB image targets using a template matching paradigm, characterized in that, include: S1, Obtain a monocular query image containing the target object and a pre-built reference image database; S2, obtain at least one reference image of the target object from the reference image database, and process the monocular query image and the reference image through a feature extraction network with shared parameters to obtain the corresponding query features and reference features; S3, Perform feature embedding processing on the query feature and the reference feature respectively to obtain embedded features; wherein, the feature embedding processing includes: instance normalization and convolution transformation; S4, Based on the embedded features, construct correlation features between the query features and the reference features at multiple different receptive field scales; S5, the correlation features obtained at different receptive field scales are fused, and based on the fused multi-scale correlation features, the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image are jointly predicted through at least two parallel prediction branches.

2. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, The procedure preceding step S1 also includes: Collect a multi-view image sequence of an object against a textured background to obtain the original image; The object binary mask corresponding to each frame of the original image is extracted using a foreground segmentation algorithm to obtain the first processed image sequence; The first processed image sequence is processed using structure-of-motion reconstruction technology to calculate the sparse 3D point cloud and the camera parameters corresponding to each frame. Perform coordinate system normalization on the camera parameters to construct a standardized coordinate system centered on the target object, and transform all original camera poses to the standardized coordinate system. Based on the transformed pose, the original image is cropped at the center and scaled to a uniform resolution to obtain a normalized reference image block, which is then stored together with the corresponding pose and camera parameters to form a target object reference image database.

3. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, The feature extraction network uses a pre-trained deep convolutional neural network as its backbone architecture and extracts multiple feature maps from the shallow, middle and deep layers of the backbone network. The shallow feature maps retain the geometric texture details of the target object, while the deep feature maps contain the semantic context information of the target object.

4. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, Step S3 includes: The input original feature tensor is subjected to the first layer of instance normalization to obtain normalized features; The normalized features are input into the convolutional layer to perform linear transformation of the feature space and compression and dimensionality reduction of the feature channel dimension; After activation by a nonlinear activation function, the embedded features are obtained through a second layer of instance normalization to achieve statistical consistency.

5. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, Step S4 includes: S401, based on adaptive pooling scale matching, performs adaptive average pooling on the normalized reference features. According to a preset set of size adjustment parameters, it dynamically compresses the spatial resolution of the reference features to a specific proportion of the original size. Using the scaled-down reference features as convolution kernels, it performs sliding window convolution on the query features to generate a pooled correlation response map. in, Indicates the location of the query feature. Pooling correlation response plot, Represents feature index , Indicates the height of the feature map of the reference image. This represents the reference feature after normalization. This represents the embedded query features. Indicates the first Size adjustment parameters corresponding to each feature; S402, based on context matching using hybrid dilated convolution, maintains the size of the reference feature and defines a set of dilation rates. For each dilation rate in the set, an inner product operation is performed with the reference feature at the query feature map location to generate a hole-related response map. in, Indicates the location of the query feature. Correlation response map of voids, Represents feature index Represents the set of expansion rates , These represent the vertical and horizontal indices within the convolution kernel space, respectively. Indicates the first Size adjustment parameters corresponding to each feature Indicates the height of the feature map of the reference image. This represents the width of the feature map in the reference image. Indicates query characteristics, This represents the reference feature after normalization. Indicates the first One expansion rate; S403 stacks all pooled correlation response maps and hollow correlation response maps generated at different feature levels along the channel dimension to form a high-dimensional hybrid correlation feature that integrates multi-scale and multi-receptive field matching information: in, This indicates the correlation map at the feature level. Feature Index Query feature location Response diagram at the location, Indicates the number of feature layers , Indicates at the feature level Feature Index Query feature location Pooling correlation response plot at the location, Indicates at the feature level Feature Index Query feature location Correlation response diagram of voids.

6. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, The fusion of the correlation features obtained at different receptive field scales includes: The geometric distance matrix is ​​calculated using the pose information of the reference image. The K-nearest neighbor algorithm is used to select the K nearest neighbor reference viewpoints on the geometric distance matrix for each reference viewpoint. Based on the selection results, a subset of relevant responses consistent with the pose of the current reference viewpoint is selected in the reference view dimension. Aggregation modeling is performed in the local pose neighborhood using 3D convolution. Finally, the maximum response in the reference view dimension is selected to generate the fused multi-scale correlation features.

7. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, The prediction branches include a scale prediction branch and a shift prediction branch; The scale prediction branch takes the fused multi-scale correlation features as input, performs global average pooling on the multi-scale correlation features to obtain the global receptive field, and then performs upsampling after convolution and activation to obtain global semantic features. The global semantic features are concatenated with the original local features in the channel dimension, and the logarithmic scale value is regressed through the output convolutional layer. Finally, the predicted scale is obtained through exponential transformation. The offset prediction branch uses an adaptive weighting mechanism based on the maximum response difference to suppress noise and obtain fused features. The fused features are then subjected to 3D convolution and max pooling, and then predicted based on the adaptive weighting mechanism and coordinate attention to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image.

8. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, The scale prediction branch takes the fused multi-scale correlation features as input, performs global average pooling on the multi-scale correlation features to obtain the global receptive field, and then upsamples the global semantic features after convolution and activation to obtain global semantic features. The global semantic features are concatenated with the original local features in the channel dimension, and the logarithmic scale value is regressed through the output convolutional layer. Finally, the predicted scale is obtained through exponential transformation, including: Global semantic features are extracted using the following formula: in, Represents global semantic features. Indicates an upsampling operation. Represents a non-linear activation function. This represents the convolution operation. This indicates a global average pooling operation. Indicates multi-scale correlation characteristics; The prediction scale is obtained based on the global semantic features and the original local features using the following formula: in, Indicates the relative scale factor. Indicates the convolution regression scale. Indicates the output convolutional layer. Represents a non-linear activation function. Indicates a feature fusion convolutional layer. Indicates multi-scale correlation characteristics. Represents global semantic features.

9. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, The offset prediction branch employs an adaptive weighting mechanism based on the maximum response difference to suppress noise and obtain fused features. These fused features are then subjected to 3D convolution and max pooling, and predicted using an adaptive weighting mechanism and coordinate attention to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameters of the target object relative to the reference image, including: The fused features are calculated using the following formula, followed by 3D convolution and max pooling: in, Indicates the characteristics after fusion. Represents the correlation response plot. Indicates adaptive weights, Represents the normalized activation function. This indicates the mean calculation operation along the feature channel dimension. Relationship response plot The maximum response value; The input features are pooled along both the horizontal and vertical directions to obtain feature vectors; attention weights are generated through convolution; the input features are multiplied by the horizontal and vertical attention weights to obtain the output features weighted and enhanced by the coordinate attention mechanism. in, This represents the output features after coordinate attention enhancement. This represents the input features of the coordinate attention module. The representation is based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the horizontal direction. Indicates based on The generated horizontal attention weights, This represents the feature vector obtained by pooling the input features along the vertical direction; The enhanced features are output through two convolution heads to obtain the object center heatmap and pixel offset map, respectively, so as to obtain the two-dimensional center position of the target object in the monocular query image and the scale parameter of the target object relative to the reference image.

10. The method for detecting unseen RGB image targets using template matching paradigm according to claim 1, characterized in that, Also includes: The network training steps involve using a total loss function that is a weighted sum of classification loss and regression loss. The classification loss is calculated using the following formula: in, Represents classification loss, This represents the total number of pixels involved in the loss calculation. Indicates the weight of positive samples. This represents a binary mask generated based on the center of the real object. This represents the network's prediction score graph. Indicates the weight of negative samples; The regression loss is calculated using the following formula: in, Indicates regression loss, A mask representing the effective area where an object exists. Indicates the forecast scale, Represents the true scale. Indicates the predicted offset. Indicates the actual offset. This indicates the prevention of local minima where the denominator is zero. This represents the square of the norm.

Images