A multi-template based image positioning method and system

Abstract

The application relates to image target region positioning, in particular to a multi-template-based image positioning method and system, which can solve the problem of poor positioning result accuracy when positioning easily-confused images. The method comprises the following steps: acquiring a plurality of reference templates corresponding to a target according to a reference image comprising the target, wherein the plurality of reference templates correspond to different positions of the target respectively; selecting one of the reference templates as a root template, and acquiring the relative position relationship between the root template and the remaining reference templates; acquiring an image pyramid of a to-be-positioned image comprising the target; determining the second positions of the remaining reference templates in the highest layer of the image pyramid according to the first position of the root template in the highest layer and the relative position relationship, and determining the first positioning region of the target in the highest layer according to the first position and the second positions; and searching the image pyramid layer by layer until the second positioning region of the target in the bottom layer of the image pyramid is searched, so as to position the target.

Description

Technical Field

[0001] This application relates to image target region localization, and more specifically, to an image localization method and system based on multiple templates. Background Technology

[0002] In the field of industrial vision imaging, it is often necessary to locate the target processing area in the image to be localized. This target processing area usually includes the target of interest to the user. In subsequent processing, operations such as foreground extraction can be performed based on the localized target processing area.

[0003] In the process of locating the target processing region in the image to be located, a feature is usually extracted from the image sample, and the feature is trained to obtain a reference template containing the feature. Then, the matching score between the reference template and each region in the image to be located is obtained. When the matching score between the reference template and a certain region reaches a threshold, it indicates that the region in the image to be located is consistent with the reference template, and the region is determined to be the target processing region.

[0004] However, some images to be located contain regions with similar features, and these images can be called easily confused images. When locating images according to the above scheme, because the features of the regions in easily confused images are similar, when determining the target processing region in the image to be located based on the matching score between the reference template and the regions in the image to be located, there may be regions with high matching scores but different detailed features that are all located as the same type of target processing region, resulting in poor accuracy of the location results when locating easily confused images. Summary of the Invention

[0005] To address the issue of poor accuracy in locating easily confused images, this application provides an image localization method and system based on multiple templates.

[0006] The embodiments of this application are implemented as follows:

[0007] The first aspect of this application provides an image localization method based on multiple templates, including:

[0008] Based on a reference image including the target, multiple reference templates corresponding to the target are obtained, and the multiple reference templates correspond to different positions of the target respectively;

[0009] Select one of the reference templates as the root template, and obtain the relative positional relationship between the root template and the other reference templates;

[0010] Obtain an image pyramid including the image of the target to be located;

[0011] By using the first position of the root template in the highest layer of the image pyramid and the relative positional relationship, the second position of the remaining reference templates in the highest layer is determined, and based on the first position and the second position, the first positioning area of ​​the target in the highest layer is determined;

[0012] Based on the first positioning area and the relative positional relationship, the image pyramid is searched layer by layer until the target is found in the second positioning area at the bottom layer of the image pyramid. The second positioning area is used to locate the target.

[0013] In some embodiments, the step of obtaining multiple reference templates corresponding to the target based on a reference image including the target includes:

[0014] Determine multiple features corresponding to different positions of the target, and perform deep learning training on the features according to the corresponding parameter settings to obtain the reference template;

[0015] The features include at least one of geometric features and regional features.

[0016] In some embodiments, in the relative positional relationship, the position coordinates of the center point of the outer rectangle of the root template are (d x1 d y1 The coordinates of the center point of the outer rectangle of the remaining reference template are (d x2 d y2 If the relative positional relationship between the center point of the outer rectangle of the remaining reference templates and the center point of the outer rectangle of the root template is:

[0017] Δd x =d x2 -d x1 Δd y =d y2 -d y1 .

[0018] In some embodiments, the step of determining the second position of the remaining reference templates in the highest layer of the image pyramid based on the first position of the root template and the relative positional relationship, and determining the first positioning region of the target in the highest layer based on the first position and the second position, includes:

[0019] Traverse and search the highest layer to determine the first position in the highest layer that matches the highest layer of the image pyramid generated by the root template;

[0020] Obtain the first coordinates of the first position, and determine the second coordinates based on the relative positional relationship. The second coordinates are used to characterize the second position.

[0021] When the second position matches the remaining reference template, the first positioning area of ​​the target in the highest layer is determined based on the first position and the second position.

[0022] In some embodiments, after obtaining the first coordinates of the first position and determining the second coordinates based on the relative positional relationship, the method further includes: obtaining an extended search area of ​​the second position in the image pyramid of this layer based on the second coordinates, and determining the second position in the extended search area.

[0023] In some embodiments, after determining the second coordinates, the step of obtaining the extended search region of the second position in the image pyramid of this layer based on the second coordinates includes:

[0024] The second coordinate of the second position is (d x d y Based on the affine transformation relationship and the current second coordinate's layer number in the image pyramid, the transformed third coordinate (d) is obtained. xf d yf The third coordinate is calculated using the following formula:

[0025] d xf =(cos(fAngle)*fScale*d x -sin(fAngle)*fScale*d y ) / Div

[0026] d yf =(sin(fAngle)*fScale*d x -cos(fAngle)*fScale*d y ) / Div

[0027] Where Div represents 2 raised to the power of CurLevel, where CurLevel is the layer number of the current second coordinate in the image pyramid; fScale represents the scaling scale, and fAngle represents the rotation angle.

[0028] The second coordinate and the third coordinate (d) xf d yf The region represented by the second position is the extended search area in the image pyramid of that layer.

[0029] In some embodiments, the step of searching the image pyramid layer by layer according to the first positioning region and the relative positional relationship until the target is found in the second positioning region at the bottom layer of the image pyramid includes:

[0030] When searching other layers in the image pyramid, the search range is the initial position obtained after passing the initial position obtained after searching the previous layer to the current layer.

[0031] When searching the image pyramid layer by layer for the root template and the remaining reference templates, the reference templates corresponding to overlapping regions and regions with low matching scores are deleted.

[0032] A second aspect of this application provides a multi-template-based image localization system, comprising:

[0033] The reference template acquisition module is used to acquire multiple reference templates corresponding to the target based on a reference image including the target, wherein the multiple reference templates correspond to different positions of the target.

[0034] The positional relationship determination module is used to select one of the reference templates as the root template and obtain the relative positional relationship between the root template and the other reference templates among the plurality of reference templates;

[0035] An image pyramid generation module is used to acquire an image pyramid including the image of the target to be located;

[0036] The template search module is used to determine the second position of the remaining reference templates in the highest layer of the image pyramid by using the first position of the root template in the highest layer and the relative positional relationship, and to determine the first positioning area of ​​the target in the highest layer based on the first position and the second position.

[0037] The positioning area determination module is used to search the image pyramid layer by layer according to the first positioning area and the relative position relationship until the target is found in the second positioning area at the bottom layer of the image pyramid. The second positioning area is used to locate the target.

[0038] In some embodiments, the step of obtaining multiple reference templates corresponding to the target based on a reference image including the target, wherein the reference template obtaining module includes an obtaining unit, the obtaining unit being used for:

[0039] Determine multiple features corresponding to different positions of the target, and perform deep learning training on the features according to the corresponding parameter settings to obtain the reference template;

[0040] The features include at least one of geometric features and regional features.

[0041] In some embodiments, the template search module includes a first positioning region acquisition unit, which is used to determine the second position of the remaining reference templates in the highest layer of the image pyramid based on the first position of the root template and the relative positional relationship, and to determine the first positioning region of the target in the highest layer based on the first position and the second position.

[0042] Traverse and search the highest layer to determine the first position in the highest layer that matches the highest layer of the image pyramid generated by the root template;

[0043] After obtaining the first coordinates of the first position and determining the second coordinates based on the relative positional relationship, the second position is obtained in the extended search area of ​​the image pyramid in this layer. The second position is then determined in the extended search area, wherein the second coordinates are used to characterize the second position.

[0044] When the second position matches the remaining reference template, the first positioning area of ​​the target in the highest layer is determined based on the first position and the second position.

[0045] The beneficial effects of this application are: by using multiple reference templates to locate the target, it is possible to accurately locate the target in easily confused images; and by utilizing the relative positional relationship between the root template and other reference templates established during training, the second position of the other reference templates can be determined by the first position of the root template, which reduces the process of performing a full traversal search of the other root templates in the first layer of the image pyramid, and greatly facilitates the image localization process. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of easily confused images;

[0048] Figure 2 This is a flowchart illustrating a multi-template-based image localization method according to one or more embodiments of this application;

[0049] Figure 3 This is a schematic diagram based on the image to be located;

[0050] Figure 4AThis is a first schematic diagram of the root template and other reference templates obtained by the multi-template-based image localization method according to one or more embodiments of this application;

[0051] Figure 4B This is a second schematic diagram of the root template and other reference templates obtained by the multi-template-based image localization method according to one or more embodiments of this application;

[0052] Figure 4C This is a third schematic diagram of the root template and other reference templates obtained by the multi-template-based image localization method according to one or more embodiments of this application;

[0053] Figure 5 This is a flowchart illustrating the process of highlighting the determination of the first localization region in a multi-template-based image localization method according to one or more embodiments of this application.

[0054] Figure 6A This is a first schematic diagram of the easily confused image localization result obtained by the multi-template-based image localization method of one or more embodiments of this application;

[0055] Figure 6B This is a second schematic diagram of the easily confused image localization result obtained by the multi-template-based image localization method according to one or more embodiments of this application;

[0056] Figure 6C This is a third schematic diagram showing the location results of easily confused images obtained by the multi-template-based image localization method according to one or more embodiments of this application;

[0057] Figure 7 This is a schematic diagram of the structure of a multi-template-based image localization system according to one or more embodiments of this application;

[0058] Figure 8 This is a schematic diagram of the structure of a computer device according to one or more embodiments of this application. Detailed Implementation

[0059] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0060] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0061] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0062] The terms “include” and “have”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0063] When locating a region in an image, a common method is to use a single feature. After training on this feature, a reference template containing that feature is obtained. Then, the region is searched for within the image based on this reference template. This process is relatively convenient. However, when using this method to locate easily confused images, such as… Figure 1 As shown, since the features of various regions in easily confused images are similar, when searching for the localization region in the image to be localized based on the reference template, it is easy to locate multiple regions with different detailed features as the same type of localization region, resulting in poor accuracy of localization results for easily confused images.

[0064] In view of this, this application proposes a multi-template-based image localization method and system. For localization scenarios involving easily confused images, multiple reference templates are generated based on the features of different positions of the target to be localized in the easily confused image, and the relative positional relationships between each reference template are obtained. Then, based on the relative positional relationships between the reference templates, the localization region in the image to be localized is obtained in the image pyramid generated corresponding to the image to be localized. At this point, it is possible to accurately distinguish each region in the easily confused image, resulting in high localization accuracy. Moreover, the entire localization process does not require traversing and searching for each reference template, which helps to improve the convenience of localizing easily confused images. Next, the multi-template-based image localization method disclosed in this application will be described in detail.

[0065] Figure 2 This is a flowchart illustrating a multi-template-based image localization method. Figure 2 As shown, in the first aspect, the image localization method based on multiple templates specifically includes the following steps:

[0066] In step 100, multiple reference templates corresponding to the target are obtained based on the reference image including the target, and the multiple reference templates correspond to different positions of the target.

[0067] Because the image selected for localization in this application is an easily confused image, which includes multiple images of regions with different shapes and similar features, for example... Figure 3The regions labeled 1, 2, and 3 shown can all be targets to be located in easily confused images. When it is necessary to locate a specific target in an easily confused image, such as... Figure 3 The area labeled 1 shown in this step refers to the region in the reference image where the target is exactly the same as the area labeled 1. This is useful when locating multiple targets in a potentially confusing image, such as... Figure 3 At least two of the regions labeled 1, 2, and 3 are included in this step. The target in the reference image of the target is one of three regions that are completely identical to the regions labeled 1, 2, and 3, respectively. "Completely identical" means regions that are identical in size, color, and shape. For ease of description, the following embodiments will use the target to be located in an easily confused image as an example.

[0068] In some embodiments, when obtaining multiple reference templates corresponding to a target, multiple features corresponding to different positions of the target are determined, and the features include at least one of geometric features and regional features. Then, the features are trained by deep learning according to the corresponding parameter settings to obtain the reference templates.

[0069] Geometric features refer to the properties used to generate geometric figures. These features mainly include the area, perimeter, thinness, duty cycle, centroid, or invariant moments of the target in the image. Commonly used region features include the area of ​​the target region in the image, the minimum bounding rectangle of the region (length, width, and angle), the region angle, the region convexity, the region roundness, and the size of the minimum bounding circle of the region.

[0070] It is understandable that when selecting features at different locations of a target, one can choose geometric features at all locations, region features at all locations, or region features at some locations and geometric features at others. However, since target localization based on region features is not suitable for images with significant lighting differences, and target localization based on geometric features is not suitable for images with blurred template edge information, and the same region may not be fully representable by a single type of feature, the specific type of feature used needs to be determined based on the actual situation of the image where localization is easily confused.

[0071] Once the features corresponding to different locations of the target are determined, all features are used as training templates for deep learning training. In some embodiments, in order to save memory occupied by training templates, after all training templates are stored in the training container, all training templates can be saved as a single training source image. In this case, during the training process, when the required training template image is called, the corresponding region can be cropped from the training source image.

[0072] It should also be noted that before training features, parameters corresponding to different features need to be set. For example, for region features, the parameters that need to be set include: upper limit of search angle, lower limit of angle, angle step size, whether to automatically increase angle step size, lower limit of scaling, upper limit of scaling, scaling step size, whether to automatically increase scaling step size, number of pyramid layers, whether to automatically increase pyramid layers, whether to enable masking, template region, template reference point, algorithm type, and mask image. For region features, the parameters that need to be set include: whether to automatically increase noise threshold, whether to automatically increase gradient magnitude, gradient magnitude threshold, noise threshold, number of pyramid layers, whether to automatically increase pyramid layers, whether to enable masking, template region, template reference point, and algorithm type. Understandably, when setting the specific parameters of the above options, one can freely select the above options or set specific parameter values ​​based on experience, or one can select the automatic default options built into the selected training model. In this case, no further selection is required from the operator, which does not require high operator experience and helps to improve the convenience of feature training. Through the above process, multiple features can be trained using deep learning according to their corresponding parameter settings, which helps to improve the accuracy of the reference template.

[0073] The structure of a deep learning training model during feature training is generally a backbone network + a head network. The backbone network occupies a much larger volume than the head network, and different deep learning algorithms can select the same backbone network. The training and learning process is a process in which the parameters inside the entire deep learning network continuously converge.

[0074] After the features have been trained using deep learning, multiple reference templates corresponding one-to-one with the target are obtained, and the deep learning training model is saved for future use.

[0075] In step 200, one of the reference templates is selected as the root template, and the relative positional relationship between the root template and the other reference templates is obtained.

[0076] Specifically, the first reference template is generally used as the root template by default, but you can also choose any reference template from all the reference templates as the root template. For example... Figure 4A As shown, Figure 4A The gray outline in the left-hand image is... Figure 3 The root template corresponding to target 1, Figure 4A The gray frame in the right-hand image is Figure 3 The remaining reference templates corresponding to Target 1; Figure 4B The gray outline in the left-hand image is... Figure 3 The root template corresponding to target 2 in the middle, Figure 4B The gray outlines in the remaining two figures are all Figure 3 The remaining reference templates corresponding to Target 2; Figure 4C The gray outline in the left-hand image is... Figure 3 The root template corresponding to target 3 in the middle, Figure 4C The gray frame in the right-hand image is Figure 3 The remaining reference templates corresponding to objective 3. It can be understood that these remaining reference templates can be called sub-templates.

[0077] Figures 4A-4C The root template and the other reference templates are obtained by training the corresponding features. The root template can be used to determine the position of the target because it can represent the general outline of the target. The other reference templates can be used to distinguish the category of the target in easily confused images because they can represent the outline of the detailed features of the target.

[0078] In the relative positional relationship, the position coordinates of the center point of the outer rectangle of the root template are (d x1 d y1 The coordinates of the center point of the outer rectangle of the remaining reference template are (d x2 d y2 If the relative positional relationship between the center point of the outer rectangle of the other reference templates and the center point of the outer rectangle of the root template is: Δd x =d x2 -d x1 Δd y =d y2 -d y1 At this point, once the coordinates of the center point of the outer rectangle of the root template are known, the x-coordinate of the center point of the outer rectangle of the root template can be directly added to d based on the aforementioned relative positional relationship. x The ordinate of the center point of the outer rectangle of the root template plus d y This allows us to determine the position coordinates of the center points of the outer rectangles of the other reference templates; or, if the position coordinates of the center points of the outer rectangles of the other reference templates are known, then based on the above relative positional relationships, we can subtract d from the x-coordinate of the center point of the outer rectangle of the other reference template. x The ordinate of the center point of the outer rectangle of the other reference template is reduced by d. y This allows you to determine the coordinates of the center point of the circumscribed rectangle of the template.

[0079] It should be noted that when there are multiple targets in the reference image, the acquired multiple reference templates are assigned according to the target type in order to obtain the relative positional relationship between the multiple reference templates corresponding to each target.

[0080] In step 300, an image pyramid including the target image to be located is obtained.

[0081] An image pyramid is a collection of images arranged in a pyramidal manner with progressively decreasing resolution, all originating from the same original image (i.e., the image to be located, including the target). The image pyramid includes multiple layers of images, with the bottom layer being the image to be located. Images in other layers are generated by sampling the image to be located. The higher the layer, the lower the image resolution.

[0082] In step 400, the second position of the remaining reference templates in the highest level of the image pyramid is determined by the first position and relative positional relationship of the root template in the highest level of the image pyramid, and the first positioning region of the target in the highest level is determined based on the first and second positions. Figure 5 As shown, the specific steps include:

[0083] Step 410: Traverse the search to the highest level and determine the first position in the highest level that matches the highest level of the image pyramid generated by the root template.

[0084] At this point, matching is performed using Normalized Cross Correlation (NCC). This involves checking if the matching score between any region in the highest layer of the image pyramid and the highest layer of the image pyramid generated from the root template is greater than a first threshold. If the score is greater than the first threshold, the two regions are considered a match, and this region in the highest layer is determined as the first position. Otherwise, the two regions are considered a mismatch. It's understandable that because the image resolution of the highest layer is relatively low, multiple first positions are selected in the highest layer to avoid missing the precise location of the first position; that is, there are multiple first positions in the highest layer with matching scores greater than the first threshold.

[0085] Step 420: Obtain the first coordinates of the first position, and determine the second coordinates based on the relative positional relationship. The second coordinates are used to characterize the second position. In other words, the second coordinates are the coordinates of the second position.

[0086] Once the first position is determined, the coordinates of the center point of the bounding rectangle of each first position can be obtained. Then, based on the relative positional relationship, the positions of the other reference templates corresponding to the first position can be obtained. Since there are multiple first positions, there are also multiple second positions in the highest layer.

[0087] In some embodiments, to further accurately obtain the second position, after determining the second coordinates, an extended search area for the second position within the current layer of the image pyramid is obtained based on the second coordinates, and the second position is determined within the extended search area. The method for obtaining the extended search area can be determined according to the actual application scenario. For example, in some embodiments, when the second coordinates of the second position are (d... x d yWhen the second coordinate is in the image pyramid, the transformed third coordinate (d) is obtained based on the affine transformation relationship and the layer number of the current second coordinate in the image pyramid. xf d yf The third coordinate is calculated as follows:

[0088] d xf =(cos(fAngle)*fScale*d x -sin(fAngle)*fScale*d y ) / Div

[0089] d yf =(sin(fAngle)*fScale*d x -cos(fAngle)*fScale*d y ) / Div

[0090] Where Div represents 2 raised to the power of CurLevel, where CurLevel is the level of the current second coordinate in the image pyramid; fScale represents the scaling scale, and fAngle represents the rotation angle. Since there are multiple second coordinates, there are also multiple third coordinates. Therefore, the multiple second coordinates and multiple third coordinates (d...) xf d yf The region that is commonly represented is the expanded search area of ​​the second position in the image pyramid of that layer. For example, the expanded search area is the second position rotated and scaled by 3 steps. At this time, since the second coordinate can cover most of the area near the second position after the rotation and scaling process, it plays a role in expanding the search position of the second position and helps to accurately determine the second position.

[0091] In some embodiments, since the second position is determined by the first position, in order to reduce deviation, the area where the initially obtained second position is located can be shifted upward, downward, leftward, and rightward to further expand the search area of ​​the second position. For example, if the shift range of the second position is 7*7 pixels, it means that the area where the second position is located is shifted upward, downward, leftward, and rightward by 3 pixels each. In some embodiments, the areas represented by the second coordinate and the third coordinate, and the area after shifting the area where the second coordinate is located, together constitute the expanded search area of ​​the second position.

[0092] Step 430: In order to ensure the accurate determination of the second position, when the second position matches the other reference templates, determine the first positioning area of ​​the target in the highest layer based on the first position and the second position.

[0093] When multiple second positions are determined, the matching score between the second position in the highest layer and the corresponding reference template is determined again to see if it is greater than the second threshold. If it is greater than the second threshold, the two regions are considered to match, and the matched second position is retained. At this time, the region where the first position and the second position are located is the first positioning region. The first positioning region can be regarded as the initial position of the target in the image pyramid of this layer.

[0094] It is understandable that the first threshold and the second threshold are determined based on the actual application scenario, and the first threshold and the second threshold may be equal or unequal.

[0095] In step 500, the image pyramid is searched layer by layer according to the first positioning area and the relative position relationship until the target is found in the second positioning area at the bottom of the image pyramid. The second positioning area is used to locate the target.

[0096] In this process, when searching other layers of the image pyramid, the search range is the initial position obtained after passing the initial position from the previous layer to the current layer. In some embodiments, the search range of the root template and other reference templates in this layer is the area corresponding to the initial position of the root template and other reference templates in the previous layer of the image pyramid. In other embodiments, the search range of the root template and other reference templates in this layer is the area corresponding to the extended search area of ​​the root template and other reference templates in the previous layer of the image pyramid. When the bottom layer of the image pyramid is reached, the positions of each root template and each other reference template in the bottom layer are arranged from high to low according to their matching scores. The appropriate positions of the root template and other reference templates in this layer are determined according to the actual application scenario. At this time, the second positioning area formed by the positions of the root template and other reference templates can be used to locate the target.

[0097] In some embodiments, in order to reduce the amount of subsequent data processing, during the layer-by-layer search in the entire image pyramid, the reference templates corresponding to overlapping regions and the reference templates corresponding to regions with low matching scores are deleted.

[0098] The overlap value is defined as the percentage of overlap between the regions of the two reference templates after they are located in the image pyramid. In other words, the overlap is the ratio of the area of ​​the overlapping region of the two reference templates to the sum of the areas occupied by each reference template. When filtering images at each layer of the image pyramid, if the overlap between the two reference templates is greater than the overlap threshold, the result with the highest matching score is retained. The overlap threshold for each layer of the pyramid is calculated as follows:

[0099] The overlap threshold includes the actual calculated overlap threshold CurOverlop′ and the preset input overlap threshold ThOverlop. The overlap threshold CurOverlop is selected as the maximum value between the actual calculated overlap threshold and the preset input overlap threshold, i.e., CurOverlop′=ma×(CurOverlop,ThOverLop).

[0100] The actual calculation formula for the overlap threshold is as follows:

[0101] CurOverlop = CurLayers / (PymLayers + OverlapLayer), where CurOverlop is the actual calculated overlap threshold for the current layer, CurLayers is the current pyramid layer number, PymLayers is the total pyramid layer number, and OverlopLayer is an empirical value set to -1.

[0102] When the overlap between two regions is greater than the preset overlap threshold, the reference template corresponding to the result with the lower matching score is the overlapping reference template.

[0103] Using the methods described above Figure 3 The location result corresponding to target 1 is as follows Figure 6A As shown, Figure 3 The location result corresponding to target 2 is as follows Figure 6B As shown, Figure 3 The location result corresponding to target 3 is as follows Figure 6C As shown. Combined with Figures 6A-6C As can be seen, the multi-template-based image localization method in this application uses multiple reference templates to locate the target, which can accurately locate the target in easily confused images and is less likely to cause localization confusion or missed localization.

[0104] By utilizing the relative positional relationships between the root template and other reference templates established during training, the second position of the other reference templates can be determined by the first position of the root template. This reduces the need for a full traversal search of the other root templates in the first-level image pyramid, greatly facilitating the image localization process.

[0105] In a specific example, when identifying a screw in an image, the following steps can be taken:

[0106] Step S1: Based on the reference image including the target, obtain multiple reference templates corresponding to the target. The multiple reference templates correspond to different positions of the target. The target in the reference image is a screw, the first reference template is the nut part of the screw, and the second reference template is the nail part of the screw.

[0107] Step S2: Select the first reference template as the root template and obtain the relative positional relationship between the root template and the second reference template.

[0108] Step S3: Obtain an image pyramid including the image of the screw to be positioned.

[0109] Step S4: Determine the second position of the second reference template in the highest layer of the image pyramid by using the first position and relative position relationship of the root template in the highest layer, and determine the first positioning area of ​​the target in the highest layer based on the first and second positions.

[0110] Step S5: Based on the first positioning area and the relative position relationship, the image pyramid is searched layer by layer until the target is found in the second positioning area at the bottom of the image pyramid. The second positioning area is used to locate the target, which is a screw.

[0111] For specific limitations in the above steps, please refer to the specific limitations of the multi-template-based image localization method in the first aspect, which will not be repeated here.

[0112] Secondly, this application provides an image localization system based on multiple templates, such as... Figure 7 As shown, the image positioning system includes:

[0113] The reference template acquisition module is used to acquire multiple reference templates corresponding to the target based on a reference image including the target. The multiple reference templates correspond to different positions of the target.

[0114] The positional relationship determination module is used to select one of the reference templates as the root template and obtain the relative positional relationship between the root template and the other reference templates among the multiple reference templates;

[0115] The image pyramid generation module is used to acquire an image pyramid including the image of the target to be located;

[0116] The template search module is used to determine the second position of the remaining reference templates in the highest layer of the image pyramid by using the first position and relative position relationship of the root template in the highest layer, and to determine the first positioning area of ​​the target in the highest layer based on the first position and the second position.

[0117] The positioning area determination module is used to search the image pyramid layer by layer based on the first positioning area and the relative position relationship until the target is found in the second positioning area at the bottom of the image pyramid. The second positioning area is used to locate the target.

[0118] In some embodiments, the step of obtaining multiple reference templates corresponding to the target based on a reference image including the target, wherein the reference template obtaining module includes an obtaining unit, the obtaining unit being used for:

[0119] Determine multiple features corresponding to different positions of the target, and perform deep learning training on the features according to the corresponding parameter settings to obtain a reference template;

[0120] The features include at least one of geometric features and regional features.

[0121] In some embodiments, in the relative positional relationship, the position coordinates of the center point of the outer rectangle of the root template are (d x1 d y1 The coordinates of the center point of the outer rectangle of the remaining reference template are (d x2 d y2 If the relative positions of the center points of the outer rectangles of the other reference templates and the center point of the outer rectangle of the root template are:

[0122] Δd x =d x2 -d x1 Δd y =d y2 -d y1 .

[0123] In some embodiments, the template search module includes a first positioning region acquisition unit, which determines the second position of the remaining reference templates in the highest layer of the image pyramid based on the first position and relative positional relationship of the root template in the highest layer, and determines the first positioning region of the target in the highest layer based on the first and second positions. The first positioning region acquisition unit is used for:

[0124] Traverse the search at the highest level and determine the first position in the highest level that matches the highest level of the image pyramid generated by the root template;

[0125] Obtain the first coordinate of the first position, and determine the second coordinate based on the relative positional relationship. The second coordinate is used to characterize the second position.

[0126] When the second position matches the remaining reference templates, the first positioning area of ​​the target in the highest layer is determined based on the first and second positions.

[0127] In some embodiments, after obtaining the first coordinates of the first position and determining the second coordinates based on the relative positional relationship, the template search module includes an extended search region determination unit, which is used to: obtain an extended search region of the second position in the image pyramid of this layer based on the second coordinates, and determine the second position in the extended search region.

[0128] In some embodiments, after determining the second coordinates, based on the second coordinates, the step of obtaining the extended search region of the second position in the image pyramid of this layer is performed. The extended search region determination unit is used for:

[0129] The second coordinate of the second position is (d) x d y Based on the affine transformation relationship and the layer number of the current second coordinate in the image pyramid, the transformed third coordinate (d) is obtained. xf d yf The third coordinate is calculated as follows:

[0130] d xf =(cos(fAngle)*fScale*d x -sin(fAngle)*fScale*d y ) / Div

[0131] d yf =(sin(fAngle)*fScale*d x -cos(fAngle)*fScale*d y ) / Div

[0132] Where Div represents 2 raised to the power of CurLevel, where CurLevel is the level of the current second coordinate in the image pyramid; fScale represents the scaling scale, and fAngle represents the rotation angle.

[0133] Second and third coordinates (d) xf d yf The region represented is the extended search area of ​​the second position in the image pyramid of this layer.

[0134] In some embodiments, the image pyramid is searched layer by layer according to the first positioning region and the relative positional relationship until the second positioning region at the bottom layer of the image pyramid is found. The positioning region determination module includes a positioning unit, which is used for:

[0135] When searching other layers in the image pyramid, the search range is the initial position obtained after passing the initial position obtained after searching the previous layer to the current layer.

[0136] When searching for the root template and other reference templates layer by layer in the image pyramid, the reference templates corresponding to overlapping areas and the reference templates corresponding to areas with low matching scores are deleted.

[0137] For specific limitations regarding multi-template-based image localization systems, please refer to the limitations outlined above; they will not be repeated here. Each module in the aforementioned multi-template-based image localization system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, allowing the processor to invoke and execute the corresponding operations of each module.

[0138] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores multi-template-based image localization data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a multi-template-based image localization method.

[0139] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] A fourth aspect of this application provides a computer-readable storage medium storing a multi-template-based image localization program, which, when executed by a processor, implements the multi-template-based image localization method of the first aspect.

[0141] The beneficial effects of this embodiment are that by using multiple reference templates to locate the target, the target in easily confused images can be accurately located; and by utilizing the relative positional relationship between the root template and the other reference templates established during training, the second position of the other reference templates can be determined by the first position of the root template, which reduces the process of performing a full traversal search of the other root templates in the first layer of the image pyramid, and greatly facilitates the image localization process.

[0142] Furthermore, by determining the second coordinates and then, based on those coordinates, obtaining the extended search area of ​​the second position within the image pyramid of this layer, it helps to reduce the possibility of missing the second position.

[0143] By searching layer by layer on the image pyramid, only the top layer needs to be traversed. Subsequent layers only require searching the initial positions obtained from the previous layer, effectively reducing the search volume and improving localization efficiency. Furthermore, by removing reference templates corresponding to overlapping areas and areas with low matching scores, subsequent computational processing is reduced, improving the ease of localization for easily confused images.

[0144] Furthermore, by determining the second coordinates and then obtaining the extended search area of ​​the second position in the image pyramid of this layer, it is helpful to expand the search area of ​​the second position, reduce the possibility of missing the second position, and thus help improve the accuracy of locating easily confused images.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-template-based image localization method, characterized in that, The method includes: Based on a reference image including the target, multiple reference templates corresponding to the target are obtained, and the multiple reference templates correspond to different positions of the target respectively; Select one of the reference templates as the root template, and obtain the relative positional relationship between the root template and the other reference templates; Obtain an image pyramid including the image of the target to be located; By using the first position of the root template in the highest layer of the image pyramid and the relative positional relationship, the second position of the remaining reference templates in the highest layer is determined, and based on the first position and the second position, the first positioning region of the target in the highest layer is determined, including: traversing and searching the highest layer to determine the first position in the highest layer that matches the highest layer of the image pyramid generated by the root template; Obtain the first coordinates of the first position, and determine the second coordinates based on the relative positional relationship, and the second coordinates of the second position. Based on the affine transformation relationship and the current second coordinate's layer number in the image pyramid, the transformed third coordinate is obtained. The third coordinate is calculated using the following formula: ； ； in 2 is a power of CurLevel, where CurLevel is the level of the current second coordinate in the image pyramid; fScale represents the scaling scale, and fAngle represents the rotation angle. The second coordinate and the third coordinate The region represented is the extended search area of ​​the second position in the image pyramid of this layer, and the second position is determined in the extended search area; the second coordinates are used to represent the second position. When the second position matches the remaining reference templates, the first positioning area of ​​the target in the highest layer is determined based on the first position and the second position; Based on the first positioning area and the relative positional relationship, the image pyramid is searched layer by layer until the target is found in the second positioning area at the bottom layer of the image pyramid. The second positioning area is used to locate the target.

2. The image localization method based on multiple templates as described in claim 1, characterized in that, The step of obtaining multiple reference templates corresponding to the target based on a reference image including the target includes: Determine multiple features corresponding to different positions of the target, and perform deep learning training on the features according to the corresponding parameter settings to obtain the reference template; The features include at least one of geometric features and regional features.

3. The image localization method based on multiple templates as described in claim 1, characterized in that, In the relative positional relationship, the position coordinates of the center point of the circumscribed rectangle of the root template are (d x1 d y1 The coordinates of the center point of the outer rectangle of the remaining reference template are (d x2 d y2 Then, the relative positional relationship between the center point of the outer rectangle of the remaining reference templates and the center point of the outer rectangle of the root template is: △d x =d x2 -d x1 ,△d y =d y2 -d y1 。 4. The image localization method based on multiple templates as described in claim 1, characterized in that, The step of searching the image pyramid layer by layer according to the first positioning region and the relative positional relationship until the target is found in the second positioning region at the bottom layer of the image pyramid includes: When searching other layers in the image pyramid, the search range is the initial position obtained after passing the initial position obtained after searching the previous layer to the current layer. When searching the image pyramid layer by layer for the root template and the remaining reference templates, the reference templates corresponding to overlapping regions and regions with low matching scores are deleted.

5. A multi-template-based image localization system, characterized in that, include: The reference template acquisition module is used to acquire multiple reference templates corresponding to the target based on a reference image including the target, wherein the multiple reference templates correspond to different positions of the target. The positional relationship determination module is used to select one of the reference templates as the root template and obtain the relative positional relationship between the root template and the other reference templates among the plurality of reference templates; An image pyramid generation module is used to acquire an image pyramid including the image of the target to be located; The template search module is used to determine the second position of the remaining reference templates in the highest layer of the image pyramid by using the first position of the root template in the highest layer and the relative positional relationship, and to determine the first positioning area of ​​the target in the highest layer based on the first position and the second position, including: traversing the highest layer to determine the first position in the highest layer that matches the highest layer of the image pyramid generated by the root template; Obtain the first coordinates of the first position, and determine the second coordinates based on the relative positional relationship, and the second coordinates of the second position. Based on the affine transformation relationship and the current second coordinate's layer number in the image pyramid, the transformed third coordinate is obtained. The third coordinate is calculated using the following formula: ； ； in 2 is a power of CurLevel, where CurLevel is the level of the current second coordinate in the image pyramid; fScale represents the scaling scale, and fAngle represents the rotation angle. The second coordinate and the third coordinate The region represented is the extended search area of ​​the second position in the image pyramid of this layer, and the second position is determined in the extended search area; the second coordinates are used to represent the second position. When the second position matches the remaining reference template, the first positioning area of ​​the target in the highest layer is determined based on the first position and the second position; The positioning area determination module is used to search the image pyramid layer by layer according to the first positioning area and the relative position relationship until the target is found in the second positioning area at the bottom layer of the image pyramid. The second positioning area is used to locate the target.

6. The image localization system based on multiple templates as described in claim 5, characterized in that, The step of obtaining multiple reference templates corresponding to the target based on a reference image including the target, wherein the reference template obtaining module includes an obtaining unit, the obtaining unit being used for: Determine multiple features corresponding to different positions of the target, and perform deep learning training on the features according to the corresponding parameter settings to obtain the reference template; The features include at least one of geometric features and regional features.

Images