Rapid image template matching method and device, storage medium and program product

Abstract

The invention discloses a rapid image template matching method and device, a storage medium and a program product, template features and image features are extracted and normalized by using a deep learning model, the template features are made into a plurality of templates through different angles and scales, effective masks are generated to indicate effective pixel points of the templates, and the effective pixel points of the templates are matched with the effective masks. Convolution operation of each template and image features is calculated in parallel, and a matching position and a matching template are determined according to a convolution result. According to the method, the parallel calculation mode of matching is greatly improved, and the matching process is accelerated.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a fast template matching technique. Background Technology

[0002] Template matching is widely used in industrial computer vision scenarios. The problem it solves is to find, given a reference image and a template defined by the user, the location and pose angle of the target in the image that matches the template.

[0003] Template matching can be implemented in various ways, but currently it mainly falls into two basic categories: the traditional image processing approach and the deep learning approach. The basic principle of the traditional image processing approach is to extract features from both the template and the image, then perform feature comparisons by sliding a window sequentially, finding the target location on the test image with the most similar features. Clearly, this method heavily relies on the effectiveness of the extracted features. Therefore, traditional image processing research has historically invested heavily in manually designing good image features—known as handcrafted features. However, currently, no handcrafted feature can perfectly represent the characteristics of an image. With the rise of deep learning, deep learning features have significant advantages over handcrafted features, and can better represent images. The deep learning approach involves passing the image and template separately through the same pre-trained network structure to extract image features and template features respectively, and then comparing the two features. The advantage of deep learning lies in the fact that the features it extracts are far superior to those extracted by handcrafted methods. However, it also has disadvantages: First, deep learning is computationally intensive. When comparing the features extracted by the deep learning model with the template, the template needs to be rotated and scaled significantly, and a sliding window is required for comparison. These operations also exist in traditional image processing, but because handcrafted features in traditional methods are generally low-dimensional, the computational cost is not significantly higher than that of deep learning. These operations cause the computational cost to increase rapidly and exponentially, further impacting matching time. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention proposes a fast image template matching method, device, storage medium, and program product based on deep learning features.

[0005] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A fast image template matching method involves inputting a template image and the image to be matched into the same pre-trained deep learning model for feature extraction, obtaining template features and image features. The template features and image features are then normalized along the channel dimension to obtain unit-length template feature vectors and image feature vectors. The unit-length template feature vectors are used to create several templates at different angles and scales, and these templates are stacked together along the batch dimension. Simultaneously, an effective mask is generated based on the template creation process. This effective mask indicates the effective and invalid regions of the template, where invalid regions refer to areas where the template is rotated or padded. During the `ng` operation, black borders or content not originally belonging to the template generated at the image edges are set to 0 for invalid regions and 1 for valid regions outside the invalid regions, thus generating the valid mask. The convolution kernel of each template is convolved with a unit-length image feature vector, and the convolution result is divided by the number of valid pixels determined by the valid mask to obtain the final convolution result for each template. The maximum value is found in the final convolution result, and the position of this maximum value is the matching position of the template. The maximum values ​​of all templates in the final convolution result are compared to determine the best matching template and obtain the corresponding best angle and scale.

[0006] Furthermore, all pooling layers in the deep learning model structure are removed, and the stride parameter of the convolutional layer preceding the removed pooling layers is adjusted to meet the feature map size requirements.

[0007] Furthermore, when training the deep learning model, a training task using pose detection is selected.

[0008] Furthermore, the training dataset of the deep learning model is a combination of several pose keypoint datasets. Since the number of keypoints detected in each pose keypoint dataset is different, the model outputs the dataset with the largest number of keypoints. If the number of keypoints is less than the maximum number, zeros are padded during the output.

[0009] Furthermore, when training the deep learning model, if the number of keypoints detected in the current pose keypoint dataset is less than the maximum number and zero-padding is performed, the corresponding zero-padding part should be removed when calculating the output loss function.

[0010] Furthermore, when training the deep learning model, the model is optimized by calculating a loss function that combines the target location, category, and keypoints.

[0011] Secondly, the present invention also proposes a computer device, the computer device including a processor and a memory, the memory storing a computer program, and the processor for executing the computer program to implement the fast image template matching method.

[0012] Furthermore, the present invention also proposes a computer storage medium storing a computer program that, when executed on a processor, implements the fast image template matching method.

[0013] Finally, the present invention also proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implement the fast image template matching method.

[0014] The beneficial effects of adopting the above technical solution are as follows: This invention addresses the problem of low matching efficiency caused by excessive computation in template matching during deep learning processing. It implements a method for parallel similarity calculation of templates at different angles and scales. This method simulates convolution operations and employs an effective masking method to mark valid regions. The convolution operation, the calculation of the effective mask, and the determination and comparison of matching positions between templates can all be fully parallelized on CPUs and GPUs, minimizing for-loop operations in the program and thus achieving fast template matching calculations. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0017] Image features (whether handcrafted or deep learning-based) are represented by a vector. If the number of feature points on a template is M, and the length of the vector (i.e., the feature dimension) is d, then the total number of features on the template is M*d. Similarity calculation involves calculating the similarity between vectors, commonly using cosine distance, which is the angle between two vectors. This is equivalent to the dot product of two direction vectors of unit 1, obtained by taking the dot product of two feature vectors and dividing by the lengths (magnitudes) of the two vectors. This operation can be linked to convolution. Convolution essentially multiplies two sets of vectors positionally and then sums them, which is also a dot product operation. Therefore, in this invention, the feature map of the template is used as the convolution kernel, and the convolution kernel is used to perform convolution with the feature map of the image. In order to satisfy the above requirement that the dot product should be divided by the magnitude of the two vectors, the image features and template features are first normalized in dimension C (the feature size extracted by deep learning is generally B, C, H, W, where B is Batch, C is Channel, H is Height, and W is Width) to make them into unit feature vectors.

[0018] In typical deep learning models, when extracting features from a single image, the 'B' in BCHW (Browser-Chew-Warp) is set to 1. In this invention, however, 'Batch' refers to multiple templates created using different rotation angles and scales. For example, assuming the search angle range is [-30 degrees, 30 degrees], with each rotation angle at a 1-degree interval, 61 angle templates need to be created. If the scale range is [0.9, 1.2], with each scale interval at a 0.1-degree interval, 4 scale templates need to be created. Therefore, there are a total of 61 * 4 = 244 templates. These created templates are then stacked together, resulting in a batch size of 244.

[0019] Figure 1This demonstrates the data processing flow in the method of this invention, which is equivalent to demonstrating the application of the method's process. The template image and the image to be matched are processed by the same deep learning model to obtain template features and image features, respectively. These two features are normalized in the channel dimension, i.e., to obtain a direction vector of unit length. The template features are then used to create multiple templates with different angles and scales, and stacked together in the batch dimension (i.e., placed side-by-side for subsequent parallel computation). Simultaneously, based on some useless regions generated during the template creation process, an effective_mask is calculated to indicate which regions are valid and which are invalid. When the template or image undergoes rotation or other operations, black borders may appear at the edges of the image, or content that does not originally belong to the template may be added. In this case, an effective_mask is set. Content belonging to the original template is set to 1 on the effective_mask, while redundant regions added due to rotation or other padding operations are set to 0. The purpose of the effective_mask is to indicate which pixels of the current template are valid. After subsequent convolution operations, the effective pixels are averaged (i.e., the convolution result is divided by the number of valid pixels). For each template, the original image is used as the image to be convolved, and the template is used as the convolution kernel for convolution. The convolution result is divided by the number of effective pixels indicated by the effective_mask to obtain the convolution similarity result. With padding, the convolution result is a map image of the same size as the original image. Each value on the map image represents the similarity between the template and that position (H, W). Finding the maximum value on the map image reveals the similarity position of the current template in the current image. This process is repeated for all created template images to find the most similar positions to the original image at all different angles and scales. A final comparison is then performed to find the most similar position to the original image, along with its corresponding angle and scale. This convolution and comparison process is repeated for all different templates. Figure 1 As shown in the red box area, in a deep learning platform, multiple parallel convolutional layers can be set up to achieve parallel computation. With the help of GPU acceleration, better parallelism can be achieved, thus realizing the above computation process faster.

[0020] The above summarizes the main ideas and implementation of this invention. In addition, some practical situations are noteworthy and suggest that this invention can be optimized or improved. Commonly used pre-trained deep models (such as ResNet pre-trained on ImageNet) extract features that lose a significant amount of angular information because these models are mostly pre-trained for classification or object detection tasks. Taking object detection as an example, detecting a cat involves identifying the cat's position in an image and whether it is a cat. When preparing the training dataset, the cat's position and name are marked in the training images. However, the cat's pose angle is not labeled during this process. Imagine if the training set contained 10 images of the same cat, each with a different pose angle. If the model were trained using such a set, its learning ability would ignore the cat's pose angle and only focus on the cat's position. Therefore, the features extracted by such a pre-trained model will contain more features of the cat itself, but lack features related to the cat's pose angle. If this model is used to extract features from the 10 images mentioned earlier, the extracted features may be very similar. However, in template matching tasks, the pose angle of the target is also required. Therefore, such features have certain shortcomings. In addition, pooling layers in deep learning models are generally considered to have the function of compressing features and removing target pose information.

[0021] In this embodiment, the model structure that can be used is ResNet50, with all pooling layers removed. Simultaneously, to meet the network design requirements of reduced feature map size, the stride of the convolutional layer preceding the original pooling layer is set to be greater than 1, for example, 2. In fact, other deep learning models can also be used; simply follow the above method, removing the pooling layers and modifying some convolutional strides.

[0022] For the task of pre-training the model, this invention does not use classification or object detection tasks, but instead employs pose detection as the training task. Pose detection, such as human pose detection in computer vision or keypoint detection of other targets, detects not only the target's position but also its pose keypoints. Such detection tasks preserve more of the target's pose information, making it more beneficial for the model to develop the ability to extract pose information during training.

[0023] Training the model requires a large number of pre-trained images. Therefore, multiple pose keypoint datasets can be combined, including COCO Animals, Animal Pose Dataset, PoseTrack, PartNet, LineMod, ETHZ Object Tracking, MSCOCO-3D Medical, OCTA-500, 3D JointSegmentation and Tracking, YCB-Video Dataset, Plant Pose Dataset, and Insect Pose. Since the number of detected keypoints varies across datasets, the model outputs the dataset with the highest number of keypoints. For keypoints with fewer than the maximum number, zeros are padded during calculation.

[0024] During model training, the loss is calculated jointly by the target location, category, and keypoints to optimize the model. It's important to note that if the dataset lacks keypoints and zeros are padded, the zero-padding portion should be removed from the calculated loss.

[0025] Once training is complete, the core of the model can be used for feature extraction required by this invention.

[0026] The deep learning model trained through the above process improves the effectiveness of deep learning features, making the matching more accurate.

[0027] Furthermore, the present invention also relates to a computer device comprising a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the above-described fast image template matching method.

[0028] Furthermore, the present invention also relates to a computer storage medium storing a computer program that, when executed on a processor, implements the above-described fast image template matching method.

[0029] Finally, the present invention also relates to a computer program product, including a computer program or instructions that, when executed by a processor, implement the above-described fast image template matching method.

[0030] It will be understood by those skilled in the art that this invention may relate to apparatus for performing one or more of the operations described in this application. The apparatus may be specifically designed and manufactured for the desired purpose, or may include known devices in general-purpose computers with programs stored therein that can be selectively activated or reconfigured. Such computer programs may be stored in a device (e.g., computer)-readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, including but not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), random access memory (RAM), read-only memory (ROM), electrically programmable ROM, electrically erasable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic cards, or optical cards. A readable medium includes any mechanism for storing or transmitting information in a form readable by a device (e.g., computer). For example, readable media include random access memory (RAM), read-only memory (ROM), disk storage media, optical storage media, flash memory devices, signals propagated in electrical, optical, acoustic, or other forms (e.g., carrier waves, infrared signals, digital signals), etc.

[0031] Those skilled in the art will understand that each box in these structure diagrams and / or block diagrams and / or flow diagrams, as well as combinations of boxes in these structure diagrams and / or block diagrams and / or flow diagrams, can be implemented using computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing method to generate a machine, thereby creating, through execution by the processor of the computer or other programmable data processing method, methods specified in the boxes of the structure diagrams and / or block diagrams and / or flow diagrams.

[0032] It will be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that the term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

Claims

1. A fast image template matching method, characterized in that: The template image and the image to be matched are input into the same pre-trained deep learning model for feature extraction to obtain template features and image features. The template features and image features are then normalized along the Channel dimension to obtain unit-length template feature vectors and image feature vectors. The unit-length template feature vectors are used to create several templates at different angles and scales, and these templates are stacked together along the Batch dimension. Simultaneously, a valid mask is generated based on the template creation process. This valid mask indicates the valid and invalid regions of the template. The invalid regions refer to areas where the template is damaged during rotation or padding operations. For black borders or content that doesn't belong to the template, the invalid region is set to 0, and the valid region outside the invalid region is set to 1, thus generating the valid mask. The convolution kernel of each template is convolved with a unit-length image feature vector, and the convolution result is divided by the number of valid pixels determined by the valid mask to obtain the final convolution result for each template. The maximum value is found in the final convolution result, and the position of this maximum value is the matching position of the template. The maximum values ​​of all templates in the final convolution result are compared to determine the best matching template and obtain the corresponding best angle and scale.

2. The fast image template matching method according to claim 1, characterized in that: Remove all pooling layers from the deep learning model structure and adjust the stride parameter of the convolutional layer preceding the removed pooling layers to meet the feature map size requirements.

3. The fast image template matching method according to claim 1, characterized in that: When training the deep learning model, a training task using pose detection is selected.

4. The fast image template matching method according to claim 1, characterized in that: The training dataset of the deep learning model is a combination of several pose keypoint datasets. Since the number of keypoints detected in each pose keypoint dataset is different, the model outputs the dataset with the largest number of keypoints. If the number of keypoints is less than the maximum number, zeros are padded during the output.

5. The fast image template matching method according to claim 4, characterized in that: When training the deep learning model, if the number of keypoints detected in the current pose keypoint dataset is less than the maximum number and zero-padding is performed, the corresponding zero-padding part should be removed when calculating the output loss function.

6. The fast image template matching method according to claim 1, characterized in that: When training the deep learning model, the model is optimized by calculating a loss function that combines the target location, category, and keypoints.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the fast image template matching method as described in any one of claims 1-6.

8. A computer storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the fast image template matching method as described in any one of claims 1-6.

9. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by a processor, they implement the fast image template matching method as described in any one of claims 1-6.

Images