Method for searching for a positioning template image in a natural scene

Abstract

The application provides a method for searching and positioning a template image in a natural scene, which uses a pre-trained deep image representation model to respectively perform visual representation on a template image and a scene image, combines similarity measurement of the two to calculate attention of the template image in the scene image. A high-attention region in the scene image is extracted through contour discovery and threshold extraction, and a cosine similarity of a feature vector of the contour region and a feature vector of the template image is recalculated. Once the similarity exceeds a newly set threshold, it is considered that the contour region is successfully matched with the template image, and finally, a barycentric coordinate in the contour region is calculated as a positioning point of the template image in the scene image.

Description

Technical Field

[0001] This invention relates to the field of computer software, and more particularly to a method and apparatus for searching and locating template images in natural scenes. Background Technology

[0002] Open-domain general image target recognition and localization has always been a challenge in computer vision. Existing methods are often limited to the annotation and model training of specific targets, which is often time-consuming and costly. Moreover, once a new category needs to be added, the annotation and training process must be repeated.

[0003] If multi-target image recognition and localization can be achieved through post-processing using features extracted from a pre-trained general visual representation model without the need for re-annotation and training, it will significantly reduce the time required for application deployment. For example, in AR applications, it is necessary to match specific target images in a scene and trigger the display of enhanced resources. Since the template images to be matched are open-domain and frequently changed, a method is needed to quickly trigger search and localization within scene images by adding a few template images. Furthermore, the method of this invention can also be used for target tracking. For instance, to describe the movement and changes of an object, where its initial location and when it will appear in the field of view are uncertain, an image template needs to be provided for automated tracking. Summary of the Invention

[0004] To address the shortcomings and deficiencies of existing technologies, this invention provides a method for searching and locating template images in natural scenes. A pre-trained deep image representation model is used to visually represent both the template image and the scene image. The similarity between the two is then used to calculate the attention of the template image within the scene image. High-attention regions in the scene image are extracted through contour discovery and thresholding. The cosine similarity between the feature vector of the contour region and the feature vector of the template image is recalculated. Once this similarity exceeds a newly set threshold, the contour region is considered to have successfully matched the template image. Finally, the centroid coordinates within the contour region are calculated as the location point of the template image in the scene image.

[0005] Considering the issue that targets change with different visual angles in real-world scenarios, the method of this invention adapts to multiple template image inputs for searching and locating targets in scene images, achieving the task of multi-target recognition and localization without the need for labeled data to train a new prediction model.

[0006] The main objective of this invention is to propose a method for searching and locating template images in scene images using depth image representation. This aims to eliminate the need for manual annotation and training through the continuous addition and replacement of template images, thereby achieving automated multi-target recognition and localization capabilities. It also provides a visual recognition triggering mechanism for related applications. The overall solution is as follows:

[0007] 1. Template image and scene image preprocessing

[0008] Before inputting the template image and scene image into the general image representation model, some preprocessing is required, mainly to normalize the data and make it conform to the requirements of the model's input forward computation. The preprocessing of the template image and the scene image is slightly different. The template image represents an image in which the entire image is meaningful, so it will first be filled with appropriate boundaries and then scaled, while the scene image will first be scaled and then cropped at the center.

[0009] 2. General Depth Image Representation

[0010] General image representation utilizes advanced visual pre-trained representation models, such as EfficientNet-V2. These models, trained on large-scale images like ImageNet, possess excellent image representation capabilities. The optimal model is then selected based on its parameter count. The input and output of pre-trained models are relatively standardized, requiring strict adherence to the tensor data shape specifications for feature map extraction.

[0011] 3. Similarity matrix calculation

[0012] The template image and the scene image have the same shape of tensor data output by the depth image representation model. This method first averages the feature map of the template image to obtain a high-dimensional vector, which represents the template image. Then, the cosine distance between the template image and the scene feature map is calculated. After that, the scene feature map will obtain a similarity matrix. The higher the value, the higher the probability that the template image is in the same position as the scene image. Finally, the attention feature map is obtained.

[0013] 4. Feature map contour extraction and discrimination

[0014] By considering the attention of the template image within the scene image, distinct high-attention regions are identified and extracted. The features of these extracted high-attention regions are then compared with the mean feature vector of the template image to calculate cosine similarity. Finally, the mean attention, maximum attention, and the recalculated cosine similarity are used, and each exceeds a certain threshold T. I This means that the region matches the template image; otherwise, it does not.

[0015] 5. Calculation of the centroid coordinates of the profile

[0016] Assuming that the extracted image contour region satisfies the requirement of matching the template image with the location region in the scene image, then calculating the centroid coordinates of this contour that satisfies the condition can be considered as the location of the template image in the scene image. The centroid coordinates are then calculated using OpenCV to solve for the moments of the contour coordinates.

[0017] The specific technical solution adopted is as follows:

[0018] A method for searching and locating template images in a natural scene, characterized by the following steps:

[0019] Step S1: Image Preprocessing

[0020] It includes two inputs, one of which is the template image T. images One is the target to be matched, and the other is the image S continuously acquired from the scene. images The two processes involve different preprocessing steps and are calculated independently.

[0021] Step S2: Depth Image Representation

[0022] For T respectively images and S images Perform deep representation to obtain feature maps;

[0023] Step S3: Attention feature map calculation and contour extraction

[0024] After step S2, T is respectively... images and S images Perform deep representation, and use the resulting feature map to calculate the attention of the template in the scene;

[0025] Step S4: Contour identification and centroid calculation

[0026] After extracting the high-attention region contours of the scene in step S3, traverse each contour and extract the attention value (ATT) within the convex hull region of each contour. key-value And extracting the attention vector (ATT) of each point within the convex hull. key-vector For AT&T key-vector Calculate the mean vector of the region, and then use this vector to further correlate with T. images The cosine similarity T is calculated from the mean feature map vectors; the discriminant condition is ATT. key-value The mean ATT key-value-mean and maximum value ATT key-value-max The contour region is considered to satisfy T if the T value is simultaneously greater than a preset threshold. images In S images The location region is determined by taking moments from the region's contour points and then calculating the centroid coordinates. These coordinates are then enlarged according to the scale ratio of the original image and the feature map, resulting in the final T. images In S images The positioning point in the middle.

[0027] Further, in step S1, T images The preprocessing process is as follows: first, compare the image width and height, then fill with 0 values, so that T images Width and height are equal, then bilinear interpolation scaling is performed; record T. imagesThe unfilled area after scaling is used for subsequent feature slice extraction; only T is calculated. images Valid information; S images The preprocessing also compares the image width and height, and then scales the smaller side to a size equal to T. images After preprocessing, they are consistent; the larger side is determined according to S. images Scale the dimensions to match the aspect ratio, then perform center cropping, with the cropped size matching T. images Consistent.

[0028] Furthermore, in step S2, EfficientNet-V2 is used for depth image representation, and T is respectively... images and S images The normalized data is processed by EfficientNet-V2 to extract feature maps, then normalized by sigmoid function transformation, and interpolated to be scaled up to match the original input scale, in order to examine the weights and attention of different feature channels.

[0029] Furthermore, in step S3, T will first... images The mean feature map of the unfilled area is calculated to obtain the feature vector representing T. images Then, using the mean feature map vector and S respectively images Calculate the cosine similarity to obtain the feature map ATT. map and ATT map Values ​​less than 0 are assigned the value 0, resulting in ATT. map Normalized Attention Feature Map (ATT) norm-map ;

[0030] Further on ATT norm-map Perform interpolation scaling and map the values ​​from the interval [0,1] to [0,255], then superimpose them onto the preprocessed S. images Above; to reduce computational load, contour extraction is performed in ATT norm-map Above operation: First, for ATT norm-map According to the threshold T att Division, less than T att The value assigned is 0, which is greater than T. att The value is assigned to 1, and then the OpenCV findContours function is used to extract the contour.

[0031] Furthermore, an automatic detection device for the degree of corrosion of transmission towers based on convolutional neural networks is characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above for searching and locating template images in a natural scene.

[0032] Furthermore, a non-transitory computer-readable storage medium storing a computer program thereon, characterized in that: when the computer program is executed by a processor, it implements the method described above for searching and locating a template image in a natural scene.

[0033] Compared to existing technologies, this invention and its preferred solution offer a simpler overall operation, easier portability, and decomposable computation, allowing for both front-end and back-end computation. This method performs different preprocessing on the template image and scene image, fully utilizing the information from the template image for overall input, improving the extraction of effective information from the scene image for center cropping, and further enhancing information utilization by using image scaling to change different scales as input. Utilizing depth image representation to calculate the attention matrix between the template image and scene image facilitates weight visualization and observation of data correlation. Extracting high-attention regions through contours and recalculating the region cosine similarity adds a condition, aiding in determining whether template matching search and localization in the scene is satisfied. Calculating the centroid coordinates of the contour as the localization point ensures that the localization point is on the target as much as possible. Attached Figure Description

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0035] Figure 1 This is a schematic diagram of the overall implementation process of the method in an embodiment of the present invention. Detailed Implementation

[0036] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below, along with accompanying drawings, for detailed explanation:

[0037] To make the objectives and technical solutions of this invention clearer, the specific embodiments of this invention are described in further detail below, such as... Figure 1 As shown, the specific steps provided are as follows:

[0038] Step S1: Image Preprocessing

[0039] There are two inputs, one is the template image (T) images One is the target to be matched, and the other is the image continuously acquired from the scene (S). images The two preprocessing steps are different and calculated independently, as shown in the flowchart for preprocessing A and preprocessing B. images The preprocessing first compares the image width and height, then fills with 0 values, so that T images The width and height are equal, then bilinear interpolation scaling is performed. This embodiment needs to record T. images The non-zero-filled area after scaling is used for subsequent feature slice extraction; only T is calculated. images Valid information. Simages The preprocessing also compares the image width and height, and then scales the smaller one to a specified size, which is the same as T. images After preprocessing, they are consistent, while larger ones follow S. images Scale the dimensions to match the aspect ratio, then perform center cropping, with the cropped size matching T. images Same.

[0040] Step S2: Depth Image Representation

[0041] The input for depth image representation is Tensor data, and the initial data transformation varies depending on the computational graph model. This embodiment uses EfficientNet-V2 as an example. The shape of the Tensor input data is (B, C, H, W), where B represents the number of images processed at once (1 in this embodiment), C represents the number of channels in the image (3 in this embodiment, corresponding to the RGB values ​​of the image), H represents the image height, and W represents the image width. After step S1, the scale required for the model input is obtained, for example, the specific shape is (1, 3, 384, 384). Then, the data is normalized along the channel number dimension, and the mean is subtracted and divided by the variance. Normalization is to divide all data values ​​by 255, where the mean is mean=[0.485, 0.456, 0.406] and the variance is std=[0.229, 0.224, 0.225]. The Tensor input data is then processed accordingly. images and S images The normalized data is processed by EfficientNet-V2 forward computation to extract feature maps of shape (1, 1280, 12, 12).

[0042] This invention interprets the feature map output by the EfficientNet-V2 model as a (12, 12) feature map with 1280 elements calculated from the original input image. The (1, 1280, 12, 12) feature map can be normalized by the sigmoid function and then interpolated to be the same scale as the original input, i.e., (1, 1280, 384, 384). This allows us to view the weights and attention of different feature channels.

[0043] Step S3: Attention feature map calculation and contour extraction

[0044] After step S2, T is respectively... images and S images Deep representation is performed, and the resulting feature map is used to calculate the attention of the template in the scene. First, T... images The mean feature map of the unfilled area is calculated, resulting in a shape of (1, 1280), indicating that T is represented by a 1280-dimensional feature vector. images Then, using this mean feature map vector, respectively, and S imagesPerforming cosine similarity yields a feature map (ATT) with shape (12,12). map ), and for ATT map Values ​​less than 0 are assigned the value 0, resulting in ATT. map Normalized Attention Feature Map (ATT) norm-map ). For ATT norm-map The interpolation scale is scaled up to (384, 384) and the values ​​are mapped from the interval [0, 1] to [0, 255], then superimposed onto the preprocessed S. images You can view T visually from above. images In S images The attention distribution in the image is such that brighter areas indicate higher attention weights. To reduce computational cost, the contour extraction is performed at an ATT of (12, 12). norm-map The above operation. First, perform the operation on ATT. norm-map According to the threshold T att (e.g., T) att =0.3) partition, less than T att The value assigned is 0, which is greater than T. att The value is assigned to 1, and then the OpenCV findContours function is used to extract the contour.

[0045] Step S4: Contour identification and centroid calculation

[0046] After step S3, the high-attention region contours of the scene have been extracted. Each contour is then traversed, and the attention value (ATT) within the convex hull region is extracted for each contour. key-value ), and extracting the attention vector (ATT) of each point within the convex hull. key-vector ), to the ATT key-vector Calculate the mean vector of the region, and then use this vector to further correlate with T. images The cosine similarity T is calculated from the mean feature map vectors. The discrimination criterion is ATT. key-value The mean (ATT) key-value-mean ) and maximum value (ATT) key-value-max ), and the T value simultaneously meets the condition of being greater than a certain threshold (e.g., ATT). key-value-mean >0.45, ATT key-value-max If T > 0.65, then the contour region can be considered to satisfy T. images In S images The localization region is then calculated, and finally, OpenCV's Moments function is used to calculate the moments of the contour points in that region, followed by the centroid coordinates. These coordinates are then scaled up according to the ratio between the original image and the feature map, resulting in the final T. images In S images The positioning point in the middle.

[0047] The main benefit of this embodiment is that multi-target image recognition and localization does not require specific labeled data or trained models. By adding a few template images, it can quickly search and locate targets in scene images, greatly reducing manual costs and accelerating subsequent application deployment, thus improving application output efficiency. It also works immediately even with frequent template image changes and modifications. A customizable visual trigger mechanism for binding template images allows for subsequent action responses once target points in the scene are identified. This provides users with customizable target recognition and localization capabilities, making it versatile and easy to promote. Furthermore, this invention can establish multi-target positional relationships, enabling multi-target position tracking.

[0048] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0049] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0050] The logic program design scheme in the above solution provided in this embodiment can be stored in a computer-readable storage medium in the form of code, and implemented in the form of a computer program. The basic parameter information required for calculation is input through computer hardware, and the calculation result is output.

[0051] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] This invention is described with reference to methods, apparatus (devices), and computer program products according to embodiments of the invention. It should be understood that each process can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing functions specified in one or more processes.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the function specified in one or more processes.

[0054] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing a function specified in one or more processes.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0056] This patent is not limited to the above-described preferred embodiments. Anyone can derive other methods for searching and locating template images in natural scenes based on the inspiration of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A method for searching and locating template images in natural scenes, characterized in that, Includes the following steps: Step S1: Image Preprocessing It includes two inputs, one of which is the template image T. images One is the target to be matched, and the other is the image S continuously acquired from the scene. images The two processes involve different preprocessing steps and are calculated independently. Step S2: Depth Image Representation For T respectively images and S images Perform deep representation to obtain feature maps; Step S3: Attention feature map calculation and contour extraction After step S2, T is respectively... images and S images Perform deep representation, and use the resulting feature map to calculate the attention of the template in the scene; In step S3, firstly, for T images The mean feature map of the unfilled area is calculated to obtain the feature vector representing T. images Then, using the mean feature map vector and S respectively images Calculate the cosine similarity to obtain the feature map ATT. map and ATT map Values ​​less than 0 are assigned the value 0, resulting in ATT. map Normalized Attention Feature Map (ATT) norm-map ; Further on ATT norm-map Perform interpolation scaling and map the values ​​from the interval [0,1] to [0,255], then superimpose them onto the preprocessed S. images Above; to reduce computational load, contour extraction is performed in ATT norm-map Above operation: First, for ATT norm-map According to the threshold T att Division, less than T att The value assigned is 0, which is greater than T. att The value is assigned to 1, and then the OpenCV findContours function is used to extract the contour; Step S4: Contour identification and centroid calculation After extracting the high-attention region contours of the scene in step S3, traverse each contour and extract the attention value (ATT) within the convex hull region of each contour. key-value And extracting the attention vector (ATT) of each point within the convex hull. key-vector For AT&T key-vector Calculate the mean vector of the region, and then use this vector to further correlate with T. images The cosine similarity T is calculated from the mean feature map vectors; the discriminant condition is ATT. key-value The mean ATT key-value-mean and maximum value ATT key-value-max The contour region is considered to satisfy T if the T value is simultaneously greater than a preset threshold. images In S images The location region is determined by taking moments from the region's contour points and then calculating the centroid coordinates. These coordinates are then enlarged according to the scale ratio of the original image and the feature map, resulting in the final T. images In S images The positioning point in the middle.

2. The method for searching and locating template images in a natural scene according to claim 1, characterized in that: In step S1, T images The preprocessing process is as follows: first, compare the image width and height, then fill with 0 values, so that T images Width and height are equal, then bilinear interpolation scaling is performed; record T. images The unfilled area after scaling is used for subsequent feature slice extraction; only T is calculated. images Valid information; S images The preprocessing also compares the image width and height, and then scales the smaller side to a size equal to T. images After preprocessing, they are consistent; the larger side is determined according to S. images Scale the dimensions to match the aspect ratio, then perform center cropping, with the cropped size matching T. images Consistent.

3. The method for searching and locating template images in a natural scene according to claim 2, characterized in that: In step S2, EfficientNet-V2 is used for depth image representation, and T is represented separately. images and S images The normalized data is processed by EfficientNet-V2 to extract feature maps, then normalized by sigmoid function transformation, and interpolated to be scaled up to match the original input scale, in order to examine the weights and attention of different feature channels.

4. An automatic detection device for the degree of corrosion of transmission towers based on convolutional neural networks, characterized in that: The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the method for searching and locating a template image in a natural scene as described in any one of claims 1-3.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the method for searching and locating template maps in a natural scene as described in any one of claims 1-3.

Images