Template selection and template matching method based on deep learning

Abstract

The invention discloses a template selection and template matching method based on deep learning, and the method comprises the steps: inputting an initial image of a to-be-selected template; inputting the initial image into a pre-trained feature extraction network, and outputting a feature map of the initial image; calculating the feature map to obtain a score map # imgabs0 #; framing an initial selected area on the score chart # imgabs1 #; calculating by using the initial selected area and the feature map to obtain a new score map # imgabs2; the selected area and the score plot # imgabs4 # are updated on the score plot # imgabs3 #; judging whether the updated selected area meets a selection condition or not; when the updated selected area does not meet the selection condition, skipping to update the selected area and the score chart # imgabs5 #; the selection condition comprises an updating condition of the score plot # imgabs6 # or a size condition of the selected area after updating; and determining and outputting the selected template according to the selected area meeting the selection condition. The problem that an existing template selection method is not high in template selection accuracy can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a template selection and template matching method based on deep learning. Background Art

[0002] Template matching is to find and locate the position of a smaller template image in another larger image when a smaller template image is known. Template matching has long been a traditional and core task in computer vision and is the key to subsequent advanced image processing. Many image processing-based tasks, such as target recognition, target detection, image registration, visual navigation and positioning, rely on the methods used in template matching and play an important role in medical image processing, satellite remote sensing image processing and other fields.

[0003] Traditional template matching tasks are to provide templates in advance and then perform template matching tasks on the target image. However, it does not provide a method to guide operators to select templates in actual engineering applications. Faced with a large number of inspection point registration tasks in actual engineering applications, operators need to spend a lot of time and energy in selecting inspection point matching templates.

[0004] With the continuous development of deep learning in the field of image processing, traditional template matching schemes have gradually stepped down from the template matching stage, and image matching algorithms based on deep learning are constantly emerging. After a long period of research and practice by domestic and foreign researchers, a variety of template matching methods have been born. Usually, the use of pre-trained special diagnosis matching backbone networks can well extract features from the input image. Compared with manually designed feature extraction schemes, such methods can be faster and more efficient.

[0005] Traditional template matching algorithms are mostly based on extracting special areas in images, encoding their characteristics into vectors for storage, and calculating the similarity of these vectors in two images to perform image matching. These algorithms are more reliable in good image quality and environment, but they are difficult to deal with situations where there are significant changes in lighting, deformation of the matching target, and interference from image background noise. There is no way to apply them to harsh scene images in actual environments.

[0006] Traditional image matching methods lack reliable template selection methods. Using traditional image matching methods, it is difficult to resist the interference from changes in external environmental lighting and noise pollution. In addition, the existing deep learning-based template matching algorithm directly returns the results after selecting the area, which often causes large errors for some images with large offsets. Therefore, optimizing template matching is an urgent problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a template selection and template matching method based on deep learning to solve the problem of low template selection accuracy in the existing template selection method.

[0008] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0009] In a first aspect, the present invention provides a template selection method based on deep learning, comprising:

[0010] L1: Input the initial image of the template to be selected;

[0011] L2: Input the initial image in step L1 into the pre-trained feature extraction network and output the feature map of the initial image;

[0012] L3: Calculate the feature map of step L2 to obtain the score map ;

[0013] L4: Score map at step L3 Select the initial selection in the upper frame;

[0014] L5: Use the initial selection in step L4 and the feature map in step L2 to calculate a new score map ;

[0015] L6: Score map at step L5 Update the selection and score map ;

[0016] L7: Determine whether the selected area updated in step L6 meets the selection conditions; when the selected area updated in step L6 does not meet the selection conditions, jump to step L6 to update the selected area and score map ; The selection conditions include step L6 score map The update condition or the size condition of the selected area after the update in step L6;

[0017] L8: Determine and output the selected template based on the selected area that meets the selection conditions in step L7.

[0018] In the above-mentioned template selection method based on deep learning, in step L4, the score map in step L3 The upper frame includes the following before the initial selection:

[0019] L41: Partition score map The area is taken as the shielding area, and the pixel values ​​in the shielding area are 0.

[0020] In the above-mentioned template selection method based on deep learning, in step L4, the initial selection area size is ,in The value range is the score map [1 / 10,1 / 5] of the width, The value range is the score map [1 / 10,1 / 5] of height;

[0021] Score map at step L3 The initial selection area in the upper frame includes:

[0022] L42: Select score map The position of the pixel with the largest value in the selection is used as the geometric center to select the initial selection area;

[0023] or,

[0024] L42: According to the preset first threshold Pair score graph Binarization is performed to obtain a binary score map ;

[0025] Among them, the first threshold The value range of is (0, 1), and the binarization process includes:

[0026] L421: Score map The pixel value is less than the first threshold The pixel value of is replaced with 0, and the score map The pixel value is greater than or equal to the first threshold The pixel value of is replaced with 1;

[0027] L422: In the binarized score map Slide upward to select an area of ​​the initial selection size and calculate the sum of the pixel values ​​within the selected area;

[0028] L423: Select the area with the highest sum of pixel values ​​within the selected area as the initial selection area;

[0029] or,

[0030] L42: In the score graph Slide upward to select an area of ​​the initial selection size and calculate the sum of the pixel values ​​in the selected area. Select the area with the highest sum of scores as the initial selection.

[0031] In the above-mentioned template selection method based on deep learning, in step L6, the score map in step L5 Update the selection and score map include:

[0032] L61: In the score graph Select the pixel with the largest pixel value;

[0033] L62: Using the pixel point selected in step L61 as the geometric center, select an area of ​​the initial selection size;

[0034] L63: Set the pixel values ​​in the area selected in step L62 to 0;

[0035] L64: According to the score map Select pixels based on the update condition;

[0036] Among them, the score map The update condition is: score map There is a pixel value greater than the second threshold The second threshold is the default value;

[0037] According to the score graph The updating condition for selecting a pixel point includes: selecting a pixel point from the pixels meeting the updating condition;

[0038] L65: Using the pixel point selected in step L64 as the geometric center, select an area of ​​the initial selection size;

[0039] L66: Set the pixel values ​​in the area selected in step L65 to 0;

[0040] L67: Obtain the minimum circumscribed rectangle of the area where step L63 and step L66 are set to 0 as the updated selection.

[0041] In the above-mentioned template selection method based on deep learning, in step L7, judging whether the selected area updated in step L6 meets the selection condition includes:

[0042] When the score graph No matching score graph found in When the pixel point of the update condition meets the selection condition, the selected area updated in step L6 meets the selection condition;

[0043] or,

[0044] When the selected area updated in step L6 meets the size condition, the selected area updated in step L6 meets the selection condition, wherein the size condition includes: the selected area updated in step L6 is larger than the score map 1 / 10 the size.

[0045] In the above-mentioned template selection method based on deep learning, in step L7, when the selected area updated in step L6 does not meet the selection condition, jump to step L6 to update the selected area and score map include:

[0046] When the selected area updated in step L6 does not meet the selection conditions, jump to step L64, and execute steps L65, L66 and L67 in sequence to obtain the updated selected area and score map. .

[0047] In the above-mentioned deep learning-based template selection method, in step L8, according to the selected area that meets the selection conditions of step L7, determining and outputting the selected template includes:

[0048] When the size of the selected area that meets the selection conditions in step L7 is smaller than the score map When the size of the selected area is 1 / 25 of that of the initial image, the geometric center of the selected area that meets the selection conditions in step L7 is mapped to the initial image input in step L1 to obtain a mapping point in the initial image. The template is selected in the initial image with the mapping point as the geometric center. The template width is 1 / 5 of the width of the initial image, and the template height is 1 / 5 of the height of the initial image. The template is output as the selected template.

[0049] When the size of the selected area that meets the selection conditions in step L7 is greater than or equal to the score map When the size is 1 / 25 of the original image size, the selected area that meets the selection conditions in step L7 is mapped to the initial image to obtain a mapping area as the selected template, and the selected template is output.

[0050] In a second aspect, the present invention provides a deep learning-based template matching method, which uses the deep learning-based template selection method described in any one of the first aspects to select an image template for template matching.

[0051] In a third aspect, the present invention provides a template matching method based on deep learning, comprising:

[0052] U1: Input image template and image to be matched;

[0053] U2: Input the image template and the image to be matched in step U1 into the pre-trained feature extraction network respectively, and output the feature map of the corresponding image template. And the feature map of the image to be matched ;

[0054] U3: Using the feature map of step U2 and feature map Calculate the score map ;

[0055] U4: The score map of step U3 Perform normalization calculation and calculate the normalized score map The average value of the whole picture ;

[0056] U5: Select the normalized score map from step U4 The pixel with the largest value ,according to The relationship between the point value and the preset matching threshold is used to determine the matching result, and the matching content is output according to the matching result;

[0057] The matching results include: complete match, partial match and no match;

[0058] When the matching result is a complete match or a partial match, the output matching content is the configuration where the image template matches the image to be matched;

[0059] When the matching result is that the match cannot be made, the output matching content is the end of the match;

[0060] The matching threshold includes a third threshold and a fourth threshold, the third threshold and the fourth threshold satisfy ;

[0061] according to The relationship between the point value and the preset matching threshold determines the matching results, including:

[0062] like Point value minus Greater than the third threshold , the matching result is a complete match;

[0063] like Point value minus Between the third threshold With the fourth threshold The matching result is a partial match;

[0064] like Point value minus Less than the fourth threshold , the matching result is that it cannot match.

[0065] In the aforementioned deep learning-based template matching method, the image template in step U1 is selected using the deep learning-based template selection method described in any one of the first aspects;

[0066] In step U5, when the matching result is a partial match, the steps include: performing matching correction and then outputting the matching content;

[0067] The output matching content after matching correction includes:

[0068] U51: Point is the geometric center, the score map after normalization in step U4 Select an area of ​​the initial selection size;

[0069] U52: Use the area selected in step U51 and the image template in step U1 to calculate the score map ;

[0070] U53: Get the score map of step U52 The pixel with the largest value among the pixels , according to the score graph of Point and Score Plots The geometric center of the score map of The point is corrected twice to obtain the point after the second correction;

[0071] Wherein, the secondary correction includes:

[0072] Score graph and score plot The upper left corner is taken as the origin, the horizontal right direction is taken as the positive direction of the x-axis, and the vertical downward direction is taken as the positive direction of the y-axis to establish a pixel point position coordinate system;

[0073] U531: Get the score map of step U52 The geometric center coordinates are expressed as , score graph The coordinates of the Q point position are expressed as , and the step U5 score map of The point position coordinates are expressed as ;

[0074] U532: Output the coordinates of the point position after secondary correction as ;

[0075] U54: Output the point after the second correction in step U53 and map it to the position of the image to be matched in step U1, indicating that the image template in step U1 is matched to the position of the image to be matched.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] The present invention selects a feature-rich and unique area from a score map of feature representation of an initial image as an initial selection based on deep learning, and obtains a new score map based on the initial selection and the initial image for subsequent updating of the selection. Combining the characteristics of deep learning and traditional image matching algorithms, the selection is used to update the score map, and the selection is performed on the updated score map, and corresponding selection conditions are designed to measure the quality of the selection, so that the method of the present invention can find the template area in images with lighting changes and noise pollution, improve the accuracy of template selection, and solve the problem of low template selection accuracy of existing template selection methods.

[0078] The present invention utilizes the idea of ​​neural network for image template matching and creatively designs a method for screening and selecting template images in a given image. The template selection method of the present invention can replace manual work, select templates with obvious features and appropriate sizes, improve the accuracy of template selection and reduce the workload of personnel.

[0079] The present invention screens the results of template matching using a deep learning method, and provides three types of matching results: complete match, partial match, and no match by performing a significance test on the matching results. When the matching result is neither a complete match nor a no match, it is classified as a partial match, and a secondary correction is performed on the partial match result, thereby reducing the false alarm rate of the non-existent template matchable area caused by judging the incomplete match as a no match, and helping to increase the accuracy of determining the matching result.

[0080] The present invention provides a more accurate matching position by performing secondary correction on partial matching. The problem of positional offset at the edge of template matching using deep learning methods is optimized. When a large offset occurs in the matching image, causing part of the template to be out of the image range, the part with obvious features in the matching area is selected and repositioned in the template image to correct the positional deviation and improve the final matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is a schematic diagram of a flow chart of a template selection method based on deep learning according to Embodiment 1 of the present invention;

[0082] Figure 2 is a schematic diagram of a template matching method based on deep learning according to Embodiment 2 of the present invention;

[0083] Figure 3 is an image of a selected template in Example 1 of the present invention ;

[0084] Figure 4 It is embodiment 1 of the present invention Figure 3 image Feature Map Schematic diagram;

[0085] Figure 5 It is embodiment 1 of the present invention Figure 4 Feature Map Schematic diagram of inner product result;

[0086] Figure 6 The preliminary selection area of ​​the first embodiment of the present invention is Schematic diagram;

[0087] Figure 7 The region of Embodiment 1 of the present invention In the feature map Execute on Convolution score map Schematic diagram;

[0088] Figure 8 The score map of five selected areas obtained in Example 1 of the present invention is Schematic diagram;

[0089] Fig. 9 is the score graph of Example 1 of the present invention The final selection diagram as a template;

[0090] Fig.10 is the score graph of Example 2 of the present invention Schematic diagram;

[0091] Fig.11 is an image to be matched in Example 2 of the present invention ;

[0092] Fig.12 is an initial image 1 of a template to be selected according to the present invention;

[0093] Fig.13 is an initial image 2 to be matched according to the present invention;

[0094] Fig.14 is an initial image 3 of a template to be selected according to the present invention;

[0095] Fig.15 It is an initial image 4 to be matched in the present invention. DETAILED DESCRIPTION

[0096] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0097] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0098] Embodiment 1:

[0099] This embodiment provides a template selection method based on deep learning. Figure 1 As shown, the following steps are included:

[0100] L1: Input the initial image of the template to be selected;

[0101] L2: Input the initial image in step L1 into the pre-trained feature extraction network and output the feature map of the initial image;

[0102] L3: Calculate the feature map of step L2 to obtain the score map ;

[0103] L4: Score map at step L3 Select the initial selection in the upper frame;

[0104] L5: Use the initial selection in step L4 and the feature map in step L2 to calculate a new score map ;

[0105] L6: Score map at step L5 Update the selection and score map ;

[0106] L7: Determine whether the selected area updated in step L6 meets the selection conditions; when the selected area updated in step L6 does not meet the selection conditions, jump to step L6 to update the selected area and score map ; The selection conditions include step L6 score map The update condition or the size condition of the selected area after the update in step L6;

[0107] L8: Determine and output the selected template based on the selected area that meets the selection conditions in step L7.

[0108] The above-mentioned template selection method based on deep learning can solve the problem of low template selection accuracy of existing template selection methods.

[0109] In step L1, input Figure 3 An image of a template to be selected is shown .image The image size is a rectangle, and the image size is between 224*224 and 1080*720. The image size of the present application is represented by width*height. The image is composed of pixels. The height of the image is the number of pixels in the vertical direction of the image; the width of the image is the number of pixels in the horizontal direction of the image. In this embodiment, the image The size is 704*576.

[0110] This application cannot extract effective features for a featureless environment such as a white wall, so the input image It cannot be a single color block.

[0111] In step L2, the image in step L1 is Input to the pre-trained feature extraction network , output image Feature map ,like Figure 4 shown.

[0112] In this embodiment, the image Send it to the vgg16 network with pre-trained weights under the pytorch framework, remove the subsequent fully connected layer part, and intercept the final output result of the convolutional layer as the feature map , feature map The tensor is represented as [1,256,H,W]. The aforementioned tensor represents a four-dimensional tensor of shape (B, C, H, W), where B is the batch number, indicating the number of images processed simultaneously; C is the number of channels, indicating the number of feature maps generated; H is the height of the feature map; and W is the width of the feature map. The feature map here The height and width are respectively the input image 1 / n of the height and width, where n is a positive integer depending on the structure of the feature extraction network N. In this embodiment, a single image I is input and processed by a neural network to generate 256 feature maps. When the image I is converted into a four-dimensional tensor, B=1, C=256, and the intercepted feature map The width of the input image 1 / 4 of the width, the intercepted feature map The height of the input image 1 / 4 of the height, H=144 (576 / 4=144), W=176 (704 / 4=176) feature map The size is 176*144.

[0113] In step L3, the feature map of step L2 is Perform inner product, such as Figure 5 The inner product result of the feature map F is shown in the tensor representation of the operation result as [1,1,H,W], and the result is normalized and processed to obtain the image Feature score plot of .

[0114] The aforementioned inner product operation refers to 256 feature maps After squaring each pixel value, each pixel value is summed along its own channel dimension, 256 feature maps Summarize into a feature map , the summarized feature map The tensor is represented as [1,1,H,W]. Before and after the inner product operation, the feature map The size remains unchanged, H=144, W=176, and the size is expressed as 176*144.

[0115] The aforementioned normalization calculation process includes:

[0116] First, get the maximum value of the 176*144 pixels in the range of 176*144, and mark the maximum value as MAX. Then divide the value of each pixel by MAX to get the score map represented by the tensor [1,1,H,W]. .

[0117] In step L4, the score map in step L3 The size of the middle box is The initial selection area, such as Figure 6 As shown, The value range is the score map [1 / 10,1 / 5] of the width, The value range is the score map The height is [1 / 10,1 / 5], and the tensor of the initial selection is expressed as [1,1,h,w]. In this embodiment, when the calculated values ​​of h and w are not integers, and when other calculation results are not integers, the calculation results follow the rounding down method. In this embodiment, For the score map 1 / 10 of the width, 176 / 10=17.6, round down, w=17; For the score map 1 / 10 of the height, 144 / 10=14.4, round down to h=14, the initial selection size is 17*14.

[0118] Before selecting the initial selection area in step L4, the following steps are included:

[0119] L41: Partition score map The area is taken as the shielding area, and the pixel values ​​in the shielding area are 0.

[0120] The edge information is insufficient and the shape and features of the edge area are often not as stable as those of the central area. Dividing the area as a shielding area can prevent the image edge from being used as a template to affect the accuracy of subsequent template matching.

[0121] In this embodiment, the method for dividing the shielded area is as follows: dividing the score map The area between the upper edge of the height and the upper edge H / 20 downward, the score graph The area between the lower edge of the height and the lower edge upward H / 20, score chart The width of the left edge and the area between the left edge to the right W / 20, and the score map The area between the right edge of width and the right edge to the left of W / 20 is used as the shielding area.

[0122] In step L4, for the score map The remaining area except the masked area can be determined using three different schemes:

[0123] L42: Select score map The position of the pixel with the largest value is used as the geometric center to select an initial selection area with a width of 17 and a height of 14;

[0124] or,

[0125] L42: According to the preset first threshold Pair score graph Binarization is performed to obtain a binary score map ;

[0126] Among them, the first threshold The value range of is (0, 1). In this embodiment , the binarization process includes:

[0127] L421: Score map The pixel values ​​of the pixels in the score map are less than the first threshold value 0.7, and the values ​​are replaced with 0; For pixels whose values ​​are greater than or equal to the first threshold value 0.7, the values ​​are replaced by 1;

[0128] L422: In the binarized score map Slide up to select a 17*14 area and calculate the sum of the pixel values ​​in the selected area;

[0129] L423: Select the area with the highest sum of pixel values ​​within the selected area as the initial selection area;

[0130] or,

[0131] In the score graph Slide up to select a 17*14 size box and calculate the sum of the pixel values ​​in the selected area. Select the area with the highest sum of pixel values ​​in the selected area as the initial selection.

[0132] Score graph The upper left corner is the origin, the horizontal right direction is the positive direction of the x-axis, and the vertical downward direction is the positive direction of the y-axis to establish the pixel point position coordinate system; Using Coordinates It indicates that, Indicates the selection area The coordinates of the upper left pixel point, Indicates the selection area The coordinates of the lower right pixel.

[0133] The images involved in this application all use the above method to establish the pixel point position coordinate system.

[0134] In step L5, in the feature map Initial selection on , initial selection In the feature map The position coordinates on , initial selection The tensor is represented as [1,256,h,w], and the initial selection In the feature map Execute on Convolution, we get Figure 7 The score graph shown , score map The tensor is represented as [1,1,H,W]. In this embodiment, the score map The size is 176*144.

[0135] Convolution is a convolution method that keeps the input and output sizes the same by padding operations. and And the sizes of the two images satisfy >> , since the image As a convolution kernel directly on the image Performing a convolution operation on the image will result in an output with the same size and image The sizes do not match. To facilitate subsequent processing, Pixels with a value of 0 are added to the outer edge of the selected area to maintain the size of the output result. In the feature map Execute on Convolution includes: In the feature map The outer edge

[0136] In this embodiment, the initial selection area In the feature map Execute on Convolution includes: In the feature map After adding the area with value 0, the initial selection As the feature map of the convolution kernel after adding the value 0 Perform convolution operation.

[0137] Feature Map The added areas with a value of 0 include: feature map The area between the upper edge of the height and h / 2 above the upper edge, feature map The area between the lower edge of the height and the lower edge H / 2 below, feature map The area between the left edge of width and the left edge w / 2, and the feature map The area between the right edge and the right edge w / 2 in width, where h=17, w=14, and the pixel value in the area with a value of 0 is 0.

[0138] In step L6, the score map The upper left corner is taken as the origin, the horizontal right direction is taken as the positive direction of the x-axis, and the vertical downward direction is taken as the positive direction of the y-axis to establish a pixel point position coordinate system;

[0139] Step L6 includes:

[0140] L61: In the score graph The coordinates of the pixel point with the largest value are expressed as ;

[0141] L62: With this pixel as the geometric center, the coordinates of the frame selection position are expressed as The area with h=17 and w=14 is selected with the pixel with the largest value in step L61 as the geometric center and a region with a height of 14 and a width of 17;

[0142] L63: Set the pixel value in the selected area to 0;

[0143] The coordinates of the upper left corner pixel of the selected area are expressed as , the coordinates of the lower right corner pixel position are expressed as In this embodiment, the position coordinates represent constituency The size is 17*14;

[0144] L64: According to the score map Select pixels based on the update condition;

[0145] Among them, the score map The update condition is: score map There is a pixel value greater than the second threshold The second threshold is the default value;

[0146] According to the score graph The updating condition for selecting a pixel point includes: selecting a pixel point from the pixels meeting the updating condition;

[0147] In this embodiment, after the update, the score map (The score graph The position coordinates are expressed as constituency The inner pixel value is 0), and the pixel value is greater than the preset second threshold. The total number of pixels that meet the value requirements is ( is a positive integer), this Select one pixel point from the pixels, and the position coordinates of the pixel point are expressed as ;

[0148] L65: Using the pixel point selected in step L64 as the geometric center, select an area of ​​the initial selection size;

[0149] The position coordinates are expressed as The pixel point is taken as the geometric center and an area with a size of 17*14 is selected;

[0150] L66: Set the pixel values ​​in the area selected in step L65 to 0;

[0151] Selection of the geometric center The position coordinates are expressed as , the coordinates of the upper left corner of the selection N2 are expressed as , the coordinates of the lower right corner are expressed as ,in, , , , In this embodiment, w=14, h=17, , , and The value of follows the principle of downward value selection;

[0152] L67: Obtain the minimum circumscribed rectangle of the area where step L63 and step L66 are set to 0 as the updated selection.

[0153] Get the position coordinates at this time as constituency The previous position coordinates are expressed as constituency The minimum enclosing rectangle of , the coordinates of the upper left corner of the minimum bounding rectangle are expressed as , the coordinates of the lower right corner are expressed as .

[0154] In step L7, it is determined whether the selected area updated in step L6 meets the selection conditions, including:

[0155] When the score graph No matching score graph found in When the pixel point of the update condition meets the selection condition, the selected area updated in step L6 meets the selection condition;

[0156] or,

[0157] When the selected area updated in step L6 meets the size condition, the selected area updated in step L6 meets the selection condition, wherein the size condition includes: the selected area updated in step L6 is larger than the score map 1 / 10 the size.

[0158] That is to judge the score map at this time The pixel value or position coordinates are expressed as Does the minimum enclosing rectangle of meet the following conditions:

[0159] Score graph Can't find a value greater than Pixels of

[0160] or,

[0161] The position coordinates are expressed as The minimum bounding rectangle size is larger than the score map 1 / 10 the size.

[0162] In step L7, when the selected area updated in step L6 does not meet the selection condition, jump to step L64, and sequentially execute steps L65, L66 and L67 to obtain the updated selected area and score map. .

[0163] In this embodiment, the selection area and score map are updated The loop execution process is as follows:

[0164] The coordinates are expressed as constituency The previous position coordinates are expressed as constituency If the minimum bounding rectangle of does not meet the selection criteria, then in the updated score map (The score graph The position coordinates are expressed as constituency The inner pixel value is 0), and the pixel value is greater than Points, select any point from the pixels that meet the requirements, and the position coordinates of the pixel point are expressed as , expressed as position coordinates The pixel point is the geometric center, select an area of ​​size 17*14, and set the pixel value in the area to 0. Selection of the geometric center The position coordinates are expressed as , the constituency The coordinates of the upper left corner are expressed as , the coordinates of the lower right corner are expressed as .

[0165] Get the position coordinates at this time as constituency , the previous position coordinates are expressed as constituency The previous position coordinates are expressed as constituency , the minimum bounding rectangle of these three selections, the position coordinates of the minimum bounding rectangle are expressed as .

[0166] Determine the score graph at this time The pixel value or position coordinates are expressed as Does the minimum enclosing rectangle of satisfy the following conditions:

[0167] Score graph Can't find a value greater than Pixels of

[0168] or,

[0169] The position coordinates are expressed as The minimum bounding rectangle size is larger than the score map 1 / 10 the size.

[0170] If the above conditions are not met, then in the updated score graph (The score graph The position coordinates are expressed as constituency The inner pixel value is 0), and the pixel value is greater than Points, select any point from the pixels that meet the requirements, and the position coordinates of the pixel point are expressed as , the pixel position coordinates are expressed as The same operation as above gives the position coordinates as constituency .

[0171] Get the current four selections ( , , and ), the minimum enclosing rectangle of which the position coordinates are expressed as .

[0172] Determine the score graph at this time The pixel value or position coordinates are expressed as Whether the minimum enclosing rectangle meets the selection criteria.

[0173] If the selection condition is not met, then in the updated score graph (The score graph The position coordinates are expressed as constituency The inner pixel value is 0), and the pixel value is greater than The pixel point position coordinates are expressed as , the above pixel position coordinates are expressed as The same operation as above gives the position coordinates as constituency .

[0174] like Figure 8 The following is a graph showing the scores of the five constituencies. Schematic diagram, get the current five selections ( , , , and ), the minimum enclosing rectangle of which the position coordinates are expressed as .

[0175] Combination Figure 8 and Fig. 9 As shown, the coordinates of the minimum circumscribed rectangle are expressed as Satisfy the size greater than the score map 1 / 10 the size.

[0176] Score graph The selection used as a template is as follows Fig. 9 As shown, the position coordinates are expressed as The constituencies are those that meet the selection criteria.

[0177] In some embodiments, updating the selection and score maps The loop execution is not a necessary process. The first execution is to step L64. The score graph There is no pixel whose value is greater than the second threshold The points of No matching score graph found in The pixel point of the update condition, the position coordinates in step L63 are expressed as constituency For eligible constituencies, That is .

[0178] In step L8, the selected template is determined and output based on the selected area that meets the selection conditions of step L7.

[0179] That is, based on Confirm and export the selected template.

[0180] The specific selection rules are as follows:

[0181] When the size of the selected area that meets the selection criteria is smaller than the score map In this embodiment, when the final selection area size is less than 1019 (177*144 / 25=1019.52), the geometric center of the selection area that meets the selection conditions is mapped to the input image. In the input image The mapping point in the image is taken as the geometric center and the box selection width is the image 1 / 5 of the width (704 / 5=140.8=140), the height is the image The area with a height of 1 / 5 (576 / 5=115.2=115);

[0182] Among them, the geometric center and score map of the selection area that meets the selection conditions The relative position relationship between the mapping point and the image The relative position relationship of is the same. In this embodiment, the score map Width and height correspond to the image 1 / 4 of the width and height, when the score map and images The upper left corner is used as the origin, the horizontal right direction is the positive direction of the x-axis, and the vertical downward direction is the positive direction of the y-axis to establish the pixel point position coordinate system. The geometric center of the selected area that meets the selection conditions is mapped to the image. A mapping point is obtained, and the position coordinates of the mapping point represent a horizontal coordinate that is four times the horizontal coordinate of the geometric center of the final selection area, and a vertical coordinate that is four times the vertical coordinate of the geometric center of the selection area that meets the selection conditions.

[0183] When the size of the selected area that meets the selection criteria is not smaller than the score map When the size is 1 / 25, the selected area that meets the selection criteria is directly mapped to the input image. The mapping area is the template In this embodiment, the final selection area size is not less than the score map 1 / 25 of the size, template The tensor representation of .

[0184] It is worth noting that the template selection method of the present embodiment 1 can be independently applied to template selection occasions or can be applied to template matching occasions in combination with the template matching method.

[0185] Embodiment 2:

[0186] Based on the same inventive concept as in Example 1, Figure 2 As shown, this embodiment introduces a template matching method based on deep learning, including the following steps:

[0187] U1: Input image template and image to be matched;

[0188] U2: Input the image template and the image to be matched in step U1 into the pre-trained feature extraction network respectively, and output the feature map of the corresponding image template. And the feature map of the image to be matched ;

[0189] U3: Using the feature map of step U2 and feature map Calculate the score map ;

[0190] U4: The score map of step U3 Perform normalization calculation and calculate the normalized score map The average value of the whole picture ;

[0191] U5: Select the normalized score map from step U4 The pixel with the largest value , judging the matching result according to the relationship between the value of point P and the preset matching threshold, and outputting the matching content according to the matching result;

[0192] The matching results include: complete match, partial match and no match;

[0193] When the matching result is a complete match or a partial match, the output matching content is the configuration where the image template matches the image to be matched;

[0194] When the matching result is that the match cannot be made, the output matching content is the end of the match;

[0195] The matching threshold includes a third threshold and a fourth threshold, the third threshold and the fourth threshold satisfy ;

[0196] according to The relationship between the point value and the preset matching threshold determines the matching results, including:

[0197] like Point value minus Greater than the third threshold , the matching result is a complete match;

[0198] like Point value minus Between the third threshold With the fourth threshold The matching result is a partial match;

[0199] like Point value minus Less than the fourth threshold , the matching result is that it cannot match.

[0200] In step U1, input image template He Ru Fig.11 The image to be matched is shown ;

[0201] Image Templates The template selection method based on deep learning described in Example 1 is used for selection;

[0202] image The image size is between 224*224 and 1080*720. In this embodiment, the image The size is 704*576.

[0203] In step U2, the image template Send it to the vgg16 network with pre-trained weights under the pytorch framework to output the feature map of the image template ;

[0204] Image Send it to the vgg16 network with pre-trained weights under the pytorch framework, intercept the final output of the convolutional layer, and output it as the feature map of the image to be matched , feature map The size is 176*144.

[0205] In step U3, the template image result feature map The resulting feature map of the image to be matched Execute on Convolution, we get Fig.10 The score graph shown In this embodiment, the score map The size is 176*144.

[0206] In step U4, the score map of step U3 is Normalized calculation, and calculated the normalized score map The average value of the whole picture .

[0207] Score graph Normalization includes: first obtaining the maximum value within the 176*144 size range and marking the maximum value as , and then the value of each pixel is divided by , and the tensor representation is [1,1,144,176] output result.

[0208] Calculating the mean of the entire graph includes: Sum the values ​​of all pixels in , and then divide the sum by the score map The total number of pixels in the whole image is obtained by the average value of the whole image. In this embodiment, the score map The total number of pixels in is 176*144.

[0209] In step U5, when the matching result is a partial match, the process includes: performing matching correction and then outputting the matching content.

[0210] The output matching content after matching correction includes:

[0211] U51: Point is the geometric center, the score map after normalization in step U4 In this embodiment, the initial selection size is selected from Point is the geometric center, the score map after normalization in step U4 Select an area of ​​size 17*14;

[0212] U52: Use the area selected in step U51 and the image template in step U1 to calculate the score map In this embodiment, the 17*14 area selected in step U51 is placed in the image template of step U1. On Convolution, get the score map ;

[0213] U53: Get the score map of step U52 The pixel with the largest value among the pixels , according to the score graph of Point and Score Plots The geometric center of the score map of The points are corrected twice to obtain the points after the second correction, so as to achieve more accurate position matching;

[0214] U54: Output the point after the second correction in step U53 and map it to the position of the image to be matched in step U1, indicating that the image template in step U1 is matched to the position of the image to be matched.

[0215] The secondary correction in step U53 includes:

[0216] Score graph and score plot The upper left corner is taken as the origin, the horizontal right direction is taken as the positive direction of the x-axis, and the vertical downward direction is taken as the positive direction of the y-axis to establish a pixel point position coordinate system;

[0217] U531: Get the score map of step U52 The geometric center coordinates are expressed as , score graph of The point position coordinates are expressed as , and the step U5 score map of The point position coordinates are expressed as ;

[0218] U532: Output the coordinates of the point position after secondary correction as .

[0219] The secondary correction operation of the present invention is to solve the problem that when the matching point is at the edge, The zero-value padding introduced by the operation causes the matching point position to be offset.

[0220] The template matching method of this embodiment uses a feature map during matching. Compared with the original image, the feature map has better feature representation capability, so it can be used for secondary matching. After the first matching obtains a result, it is equivalent to finding an area in the image to be matched that is the same size as the template image, which is recorded as area . When matching once, the template image is used In the image to be matched Match on to get the area , and the area during the second matching In the template The matching is performed on the points, and the points used for matching are corrected twice during the secondary matching. The secondary correction gives a deviation value for position correction, and finally the final matching content can be output after translation according to the deviation value.

[0221] like Fig.12 An initial image 1 of a template to be selected as shown and Fig.13 The initial image 2 to be matched shown in FIG. 2 has matching content within a green frame. The method of Embodiment 1 of the present invention can select Fig.12 The matching content in the green box is used as a template , in the method of Example 2, input Fig.13 The image shown is able to output the matching content within the green box.

[0222] When the external environment light changes, such as Fig.14 and Fig.15 The ambient lighting changes. Fig.14 The initial image 3 of the template to be selected is shown as Fig.15The initial image 4 to be matched shown in FIG. 4 has matching content within the green frame. The method of Embodiment 1 of the present invention can still select Fig.14 The matching content in the green box is used as a template , in the method of Example 2, input Fig.15 The image shown can still output the matching content within the green box.

[0223] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0224] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0227] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A template selection method based on deep learning, characterized in that: include: L1: Input the initial image of the template to be selected; L2: Input the initial image in step L1 into the pre-trained feature extraction network and output the feature map of the initial image; L3: Calculate the feature map of step L2 to obtain the score map ; L4: Score map at step L3 Select the initial selection in the upper frame; L5: Use the initial selection in step L4 and the feature map in step L2 to calculate a new score map ; L6: Score map at step L5 Update the selection and score map ; L7: Determine whether the selected area updated in step L6 meets the selection conditions; when the selected area updated in step L6 does not meet the selection conditions, jump to step L6 to update the selected area and score map ; The selection conditions include step L6 score map The update condition or the size condition of the selected area after the update in step L6; L8: Determine and output the selected template based on the selected area that meets the selection conditions in step L7.

2. The template selection method based on deep learning according to claim 1, characterized in that: In step L4, the score map in step L3 The upper frame includes the following before the initial selection: L41: Partition score map The area is taken as the shielding area, and the pixel values ​​in the shielding area are 0.

3. The template selection method based on deep learning according to claim 1 or 2, characterized in that: In step L4, the initial selection area size is ,in The value range is the score map [1 / 10,1 / 5] of the width, The value range is the score map [1 / 10,1 / 5] of height; Score map at step L3 The initial selection area in the upper frame includes: L42: Select score map The position of the pixel with the largest value in the selection is used as the geometric center to select the initial selection area; or, L42: According to the preset first threshold Pair score graph Binarization is performed to obtain a binary score map ; Among them, the first threshold The value range of is (0, 1), and the binarization process includes: L421: Score map The pixel value is less than the first threshold The pixel value of is replaced with 0, and the score map The pixel value is greater than or equal to the first threshold The pixel value of is replaced with 1; L422: In the binarized score map Slide upward to select an area of ​​the initial selection size and calculate the sum of the pixel values ​​within the selected area; L423: Select the area with the highest sum of pixel values ​​within the selected area as the initial selection area; or, L42: In the score graph Slide upward to select an area of ​​the initial selection size and calculate the sum of the pixel values ​​in the selected area. Select the area with the highest sum of scores as the initial selection.

4. The template selection method based on deep learning according to claim 1 or 3, characterized in that: In step L6, the score map in step L5 Update the selection and score map include: L61: In the score graph Select the pixel with the largest pixel value; L62: Using the pixel point selected in step L61 as the geometric center, select an area of ​​the initial selection size; L63: Set the pixel values ​​in the area selected in step L62 to 0; L64: According to the score map Select pixels based on the update condition; Among them, the score map The update condition is: score map There is a pixel value greater than the second threshold The second threshold is the default value; According to the score graph The updating condition for selecting a pixel point includes: selecting a pixel point from the pixels meeting the updating condition; L65: Using the pixel point selected in step L64 as the geometric center, select an area of ​​the initial selection size; L66: Set the pixel values ​​in the area selected in step L65 to 0; L67: Obtain the minimum circumscribed rectangle of the area where step L63 and step L66 are set to 0 as the updated selection.

5. The template selection method based on deep learning according to claim 4 is characterized in that: In step L7, determining whether the selected area updated in step L6 meets the selection conditions includes: When the score graph No matching score graph found in When the pixel point of the update condition meets the selection condition, the selected area updated in step L6 meets the selection condition; or, When the selected area updated in step L6 meets the size condition, the selected area updated in step L6 meets the selection condition, wherein the size condition includes: the selected area updated in step L6 is larger than the score map 1 / 10 the size.

6. The template selection method based on deep learning according to claim 4 or 5, characterized in that: In step L7, when the selected area updated in step L6 does not meet the selection conditions, jump to step L6 to update the selected area and score map include: When the selected area updated in step L6 does not meet the selection conditions, jump to step L64, and execute steps L65, L66 and L67 in sequence to obtain the updated selected area and score map. .

7. The template selection method based on deep learning according to claim 1, characterized in that: In step L8, according to the selected area that meets the selection conditions of step L7, the selected template is determined and outputted, including: When the size of the selected area that meets the selection conditions in step L7 is smaller than the score map When the size of the selected area is 1 / 25 of that of the initial image, the geometric center of the selected area that meets the selection conditions in step L7 is mapped to the initial image input in step L1 to obtain a mapping point in the initial image. The template is selected in the initial image with the mapping point as the geometric center. The template width is 1 / 5 of the width of the initial image, and the template height is 1 / 5 of the height of the initial image. The template is output as the selected template. When the size of the selected area that meets the selection conditions in step L7 is greater than or equal to the score map When the size is 1 / 25 of the original image size, the selected area that meets the selection conditions in step L7 is mapped to the initial image to obtain a mapping area as the selected template, and the selected template is output.

8. A deep learning template matching method, characterized in that: An image template for template matching is selected using the deep learning-based template selection method described in any one of claims 1 to 7.

9. A template matching method based on deep learning, characterized in that: include: U1: Input image template and image to be matched; U2: Input the image template and the image to be matched in step U1 into the pre-trained feature extraction network respectively, and output the feature map of the corresponding image template. And the feature map of the image to be matched ; U3: Using the feature map of step U2 and feature map Calculate the score map ; U4: The score map of step U3 Perform normalization calculation and calculate the normalized score map The average value of the whole picture ; U5: Select the normalized score map from step U4 The pixel with the largest value , judging the matching result according to the relationship between the value of point P and the preset matching threshold, and outputting the matching content according to the matching result; The matching results include: complete match, partial match and no match; When the matching result is a complete match or a partial match, the output matching content is the configuration where the image template matches the image to be matched; When the matching result is that the match cannot be made, the output matching content is the end of the match; The matching threshold includes a third threshold and a fourth threshold, the third threshold and the fourth threshold satisfy ; The matching result is determined based on the relationship between the value of point P and the preset matching threshold value, including: like Point value minus Greater than the third threshold , the matching result is a complete match; like Point value minus Between the third threshold With the fourth threshold The matching result is a partial match; like Point value minus Less than the fourth threshold , the matching result is that it cannot match.

10. The template matching method based on deep learning according to claim 9, characterized in that: The image template in step U1 is selected using the deep learning-based template selection method described in any one of claims 1 to 7; In step U5, when the matching result is a partial match, the steps include: performing matching correction and then outputting the matching content; The output matching content after matching correction includes: U51: Point is the geometric center, the score map after normalization in step U4 Select an area of ​​the initial selection size; U52: Use the area selected in step U51 and the image template in step U1 to calculate the score map ; U53: Get the score map of step U52 The pixel with the largest value among the pixels , according to the score Figure S4 Point and Score Plots The geometric center of the score map Perform secondary correction on point P to obtain the point after secondary correction; Wherein, the secondary correction includes: Score graph and score plot The upper left corner is taken as the origin, the horizontal right direction is taken as the positive direction of the x-axis, and the vertical downward direction is taken as the positive direction of the y-axis to establish a pixel point position coordinate system; U531: Get the score map of step U52 The geometric center coordinates are expressed as , score graph of The point position coordinates are expressed as , and the step U5 score map of The point position coordinates are expressed as ; U532: Output the coordinates of the point position after secondary correction as ; U54: Output the point after the second correction in step U53 and map it to the position of the image to be matched in step U1, indicating that the image template in step U1 is matched to the position of the image to be matched.

Images