Sub-pixel template matching method and device based on non-linear optimization

Abstract

The embodiment of the application discloses a kind of subpixel template matching method and device based on nonlinear optimization, method includes: obtaining template image and target image;Respectively, edge feature point extraction is carried out, and template feature point set and target feature point set are obtained, and gradient intensity graph is constructed;Based on feature point set, rough matching is determined initial position and initial angle, and current matching pose is formed by combining position offset, angle offset, and the coordinate transformation of template feature point set is carried out;The gradient intensity value of mapping point after transformation is obtained in gradient intensity graph by interpolation method;With position offset and angle offset as optimization variable, with gradient intensity value summation maximization as optimization target, iteration update is carried out, and the target position and angle of subpixel level are output.The application is simultaneously optimized position and angle by interpolation method combined with nonlinear optimization, avoids the dependence of traditional curved surface fitting on local smoothness, improves matching precision and anti-noise robustness, and can meet the demand of high-precision industrial vision positioning.

Description

Technical Field

[0001] This application relates to the field of general image data processing or generation technology, and in particular to a subpixel template matching method and apparatus based on nonlinear optimization. Background Technology

[0002] Template matching is widely used in industrial automation positioning, printed circuit board inspection, semiconductor wafer alignment and other scenarios, and sub-pixel accuracy has become a core requirement for high-end manufacturing.

[0003] However, traditional template matching schemes based on sub-pixel surface fitting have significant drawbacks in practical applications. First, surface fitting methods assume that the local similarity response surface follows a smooth quadratic surface model. When the target edge is irregular or the target image contains high-frequency noise, the actual response surface deviates significantly from the smooth surface model, leading to a substantial increase in fitting error and a sharp decrease in accuracy. Second, surface fitting methods are only effective for regions with a single, obvious peak in the similarity response map. When the response map has multiple similar peaks, the fitting process is prone to getting trapped in local optima, causing the positioning result to deviate from the true position. Furthermore, traditional methods only optimize the position coordinates in the fine matching stage. When the target has both positional and angular offsets, it is necessary to process them step by step or estimate the angle additionally. Step-by-step optimization is prone to error accumulation and cannot fully utilize the overall information of edge features, making it difficult to obtain high-precision position and angle information simultaneously in complex industrial scenarios. Summary of the Invention

[0004] This application provides a subpixel template matching method and apparatus based on nonlinear optimization, which aims to simultaneously optimize position and angle by combining interpolation with nonlinear optimization, avoiding local smoothness dependence, preventing step-by-step error accumulation, and improving matching accuracy and noise robustness.

[0005] In a first aspect, embodiments of this application provide a sub-pixel template matching method based on nonlinear optimization, including: Obtain the template image and the target image; Edge feature points are extracted from the template image and the target image respectively to obtain the template feature point set and the target feature point set. Then, a gradient intensity map is constructed to characterize the gradient intensity at each pixel position in the target image. Coarse matching is performed based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image; Based on the current matching pose, which is composed of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset, the template feature point set is transformed by coordinate transformation, and the initial values ​​of the position offset and the angle offset are zero. The gradient intensity values ​​at multiple mapping points after coordinate transformation are obtained from the gradient intensity map using interpolation. Using position offset and angle offset as optimization variables and maximizing the sum of gradient intensity values ​​as the optimization objective, the position offset and angle offset are iteratively updated until the convergence condition is met, and the sub-pixel level target position and target angle are output.

[0006] Secondly, embodiments of this application provide a sub-pixel template matching device based on nonlinear optimization, comprising: The image acquisition unit is used to acquire the template image and the target image; The feature gradient generation unit is used to extract edge feature points from the template image and the target image respectively, to obtain the template feature point set and the target feature point set, and to construct a gradient intensity map, which is used to characterize the gradient intensity at each pixel position in the target image. The initial matching unit is used to perform coarse matching based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image; The pose transformation unit is used to perform coordinate transformation on the template feature point set according to the current matching pose consisting of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset. The initial values ​​of the position offset and the angle offset are zero. The gradient interpolation unit is used to obtain the gradient intensity values ​​at multiple mapped points after coordinate transformation in the gradient intensity map by interpolation. The optimized output unit is used to iteratively update the position and angle offsets with the optimization variables and the optimization objective of maximizing the sum of gradient intensity values, until the convergence condition is met, and outputs the target position and target angle at the sub-pixel level.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps as described in the first aspect of embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps in the first aspect of embodiments of this application.

[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement some or all of the steps described in the first aspect of embodiments of this application.

[0010] As can be seen, in this embodiment, the gradient intensity value at the mapping point is obtained on the gradient intensity map by interpolation, which replaces the traditional surface fitting and eliminates the dependence on the smoothness of the local surface; the optimization objective is to maximize the sum of gradient intensity values ​​instead of maximizing the similarity function, which enhances the anti-interference ability to changes in illumination; at the same time, the position offset and angle offset are optimized to avoid the error accumulation caused by step-by-step optimization, which can obtain high-precision position and angle information in complex industrial scenarios at one time, significantly improving the accuracy and robustness of template matching. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a visual positioning system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a subpixel template matching method based on nonlinear optimization provided in an embodiment of this application; Figure 3 This is a schematic diagram of a non-maximum suppression screening scenario provided in an embodiment of this application; Figure 4 This is a schematic diagram of a coarse matching process provided in an embodiment of this application; Figure 5 This is a schematic diagram of a fine matching process provided in an embodiment of this application; Figure 6 This is a functional unit block diagram of a sub-pixel template matching device based on nonlinear optimization provided in an embodiment of this application; Figure 7 This is a functional unit block diagram of another sub-pixel template matching device based on nonlinear optimization provided in this application embodiment; Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0014] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but in some embodiments includes steps or units not listed, or in some embodiments includes other steps or units inherent to these processes, methods, products, or apparatuses.

[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] In the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist simultaneously; B exists alone. Among them, A and B can be singular or plural.

[0017] In this embodiment, the symbol " / " can indicate that the preceding and following objects are in an "or" relationship. Alternatively, the symbol " / " can also represent a division sign, i.e., performing a division operation. For example, A / B can mean A divided by B.

[0018] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0019] In the embodiments of this application, "equal to" can be used with "greater than" and is applicable to technical solutions used when "greater than" is used; it can also be used with "less than" and is applicable to technical solutions used when "less than" is used. When "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a visual positioning system provided in an embodiment of this application. Figure 1 As shown, the visual positioning system 10 includes an image acquisition device 110 and a visual processing server 120. The image acquisition device 110 is an industrial camera with image acquisition function, including terminal devices such as area scan cameras and line scan cameras, which are used to acquire image data of the target to be positioned in real time.

[0021] Specifically, the vision processing server 120 is deployed in the industrial field and has image processing and accelerated computing capabilities. To achieve high-precision sub-pixel level visual positioning, the vision processing server 120 and the image acquisition device 110 establish a communication connection through a network to form an "image acquisition-visual positioning" processing system.

[0022] In the visual positioning system 10, the image acquisition device 110 is used to acquire the target image and transmit it to the visual processing server 120. The visual processing server 120, as the core processing unit, is used to acquire the template image and the target image, extract edge feature points from the template image and the target image respectively, obtain the template feature point set and the target feature point set, construct a gradient intensity map, and perform coarse matching based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image. Based on the current matching pose composed of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset, the template feature point set is subjected to coordinate transformation. The gradient intensity values ​​at multiple mapping points after coordinate transformation are obtained in the gradient intensity map by interpolation. The position offset and the angle offset are used as optimization variables, and the sum of the gradient intensity values ​​is used as the optimization objective for iterative updates until the convergence condition is met, and the sub-pixel level target position and target angle are output.

[0023] Based on this, the embodiments of this application provide a sub-pixel template matching method based on nonlinear optimization. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0024] Please see Figure 2 , Figure 2 This is a flowchart illustrating a sub-pixel template matching method based on nonlinear optimization provided in an embodiment of this application. The method is applied to the visual processing server 120 in the visual positioning system 10, and includes: Step S201: Obtain the template image and the target image.

[0025] Here, the template image refers to a pre-selected reference image containing the target object to be located, used as a baseline template for matching; the target image refers to an industrial site image to be searched and located, acquired in real time by an image acquisition device. The design principle of this step is to provide image data sources for subsequent matching processing. The template image provides prior shape and structural information of the target object, while the target image provides the spatial range to be searched.

[0026] Step S202: Extract edge feature points from the template image and the target image respectively to obtain the template feature point set and the target feature point set, and construct a gradient intensity map.

[0027] Here, edge feature points refer to pixel locations in the image where grayscale changes drastically, typically corresponding to the contours or boundaries of the target, and can represent the geometric structure information of the target in a compact manner; the gradient intensity map is a two-dimensional matrix of the same size as the target image, used to represent the gradient intensity at each pixel location in the target image. For example, before step S202, the template image and the target image can be preprocessed with Gaussian filtering to eliminate noise interference. The Gaussian filter kernel size can be 3×3 or 5×5, and the standard deviation σ can be 0.5~1.5.

[0028] The design principle of this step is to extract the edge regions with the richest information in the image as the matching basis. Compared with directly using the original pixel grayscale, edge features have natural robustness to changes in illumination, while greatly reducing the amount of data involved in the calculation, laying the foundation for improving the efficiency and accuracy of subsequent matching.

[0029] In one possible embodiment, edge feature point extraction is performed on the template image and the target image respectively to obtain a template feature point set and a target feature point set, including: performing edge detection on the template image and the target image respectively to obtain a template edge point set and a target edge point set respectively; calculating the gradient intensity of each edge point in the template edge point set and the target edge point set respectively; performing non-maximum suppression on the edge points in the template edge point set and the target edge point set respectively, and selecting edge points whose gradient intensity is a local maximum in a preset neighborhood as the template feature point set and the target feature point set.

[0030] Edge detection refers to using edge detection operators (such as the Canny Edge Detection Operator and the Sobel Operator) to extract pixel locations in an image where grayscale changes drastically. These locations typically correspond to the contours or boundaries of a target. Gradient intensity refers to the magnitude of grayscale change at an edge point, used to measure the salience of that edge point. Non-maximum suppression refers to performing a local neighborhood search on edge points, retaining only edge points where the gradient intensity is the maximum within a preset neighborhood, and discarding the remaining non-maximum points, thereby achieving the effect of sparsifying the feature point set and suppressing noise response. The preset neighborhood refers to a local window centered on the current edge point, the size of which can be preset according to the actual application scenario.

[0031] The non-maximum suppression criterion is: the gradient intensity of the current edge point is greater than a preset edge intensity threshold, and its gradient intensity is at its maximum value within a preset neighborhood of that edge point. Specifically, for each pyramid image layer, the Canny edge detection operator is used to calculate the gradient intensity of each pixel. The gradient intensity is calculated as the square root of the sum of the squares of the horizontal and vertical gradients, as shown in the following formula: ; in, and These are the gradient components of a pixel in the horizontal and vertical directions, respectively. Then, non-maximum suppression is applied to each edge point, retaining those whose gradient intensity is a local maximum in a 3×3 neighborhood and is greater than a threshold. Edge points are identified, and a sparse feature point set is constructed. Through the above processing, template image and target image generate template feature point set and target feature point set respectively, consisting of local salient edge points, for use in the subsequent coarse matching stage.

[0032] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating a scenario for non-maximum suppression screening provided in an embodiment of this application. For example... Figure 3 As shown, the nonmaximum suppression screening process demonstrates the complete workflow of filtering local maximum feature points from the original edge points through three parts: left, middle, and right.

[0033] The left side shows the gradient intensity distribution of the original edge points. A 3×3 pixel grid with 9 cells is drawn in the image, each cell representing a pixel position in a certain layer of the target image pyramid. Each cell is filled with a different shade of gray to represent the gradient intensity at that location; the darker the gray, the higher the gradient intensity. Specifically, the first row has three light gray cells labeled with gradient intensity values ​​of 25, 30, and 22 (G=25, G=30, G=22); the second row has two light gray cells labeled with gradient intensity values ​​of 35 and 40 (G=35, G=40), and a dark gray cell labeled with a gradient intensity value of 120 (G=120); the third row has three light gray cells labeled with gradient intensity values ​​of 28, 32, and 20 (G=28, G=32, G=20). The central cell has the darkest gray level and the highest gradient intensity value, significantly higher than the surrounding 8 cells; the surrounding cells have lighter gray levels, and their gradient intensity values ​​are all at a low level between 20 and 40. The left side is used to represent the "original edge points (gradient intensity distribution)".

[0034] The middle section illustrates the screening process and decision logic for non-maximum suppression. A 3×3 pixel grid is drawn in the figure, with the central cell labeled "Retain". The symbol "" indicates that the point satisfies the criteria for a local maximum and is retained as a feature point; the surrounding 8 cells are all marked with "". The markers "Suppressed" indicate that these points are suppressed and removed because their gradient strength is not the maximum value in their neighborhood. The criteria for retaining the center point can be adapted to be: the gradient strength of the current point is 120, which is greater than the gradient strength of all points in its neighborhood (the maximum value does not exceed 45), and the gradient strength of the current point is 120, which is greater than the preset edge strength threshold of 50. If both conditions are met, the center point is determined to be a local maximum and is retained. The bottom of the middle section is marked "Non-Maximum Suppression (NMS) Filtering (Retaining Local Maximums)".

[0035] The right side shows the sparse feature point set obtained after filtering. A large rectangular area is drawn in the figure, representing a local region of the target image or a layer of the target image pyramid. Multiple hollow dots are scattered within this area, representing all edge points; only one solid black dot with a bold outer circle and a star-shaped marker is labeled "Retained Feature Point," while the remaining hollow dots are edge points that were suppressed and removed. The label "Filtering Result: Sparse Feature Point Set" on the right side indicates that the feature point set output after non-maximum suppression has been sparsified, and the text box leading to "Used for Subsequent Coarse Matching Stage" indicates that this sparse feature point set will be passed as input data to the subsequent coarse matching step.

[0036] It should be understood that the gradient intensity values ​​and thresholds mentioned above are merely illustrative examples and do not constitute any limitation on this application; the 3×3 neighborhood mentioned above is also merely an illustrative neighborhood size, and this application may also use neighborhood windows of other sizes for non-maximum suppression screening.

[0037] As can be seen, in this example, by performing non-maximum suppression filtering on edge points, only significant edge points whose gradient intensity is a local maximum within a preset neighborhood are retained as feature points. This effectively eliminates a large number of weak edges and noise points, significantly compresses the size of the feature point set, and thus reduces the computational complexity of the subsequent coarse matching and fine matching stages. At the same time, since the most significant edge structure information in the image is retained, the feature point set has strong robustness to noise interference and illumination changes, which helps to ensure the reliability of the initial pose obtained by coarse matching and provides a good starting point for the nonlinear optimization in the fine matching stage.

[0038] Step S203: Perform coarse matching based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image.

[0039] Coarse matching refers to finding the approximate position and angle of the template in the target image at a pixel-level precision scale. The initial position and initial angle are the initial matching coordinates and rotation angle at the pixel level, respectively, which are used to provide an optimized starting point for subsequent fine matching.

[0040] The design principle of this step is to obtain an initial estimate that is close to the true pose first, and then limit the search range of subsequent fine matching to a small neighborhood around the initial value, thereby avoiding the problem of excessive computation caused by sub-pixel level search over a large area.

[0041] In one possible embodiment, coarse matching is performed based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image. This includes: constructing image pyramids for the template image and the target image respectively; generating template sets with different angles and scales for the template image at each layer of the image pyramid; matching each template in the template set with the target image one by one, and performing a layer-by-layer search from the top to the bottom of the image pyramid, using the best position obtained from each layer as the search center for the next layer to obtain the initial position and initial angle.

[0042] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of a coarse matching process provided in an embodiment of this application. For example... Figure 4 As shown, the coarse matching operation process includes the following steps in sequence: Step S401: Construct image pyramids for the template image and the target image respectively.

[0043] The image pyramid refers to a multi-layered image sequence formed by downsampling the same image multiple times, from coarse to fine. The downsampling factor is 0.5, and the number of layers N is specified by the user, with a typical value of 4 to 6 layers. The top layer is the layer with the lowest resolution, and the bottom layer is the layer with the highest resolution.

[0044] Step S402: At the current pyramid level, generate a template set for the template image; perform rotation transformation within a preset angle search range by a preset angle step size; and perform scaling transformation within a preset scale search range by a preset scaling step size.

[0045] The template set refers to a collection of multiple template instances generated by rotating and scaling a template image according to a preset angle and scale range.

[0046] Specifically, on each pyramid level, the template image is rotated and scaled around its center point to generate a template set covering a preset angle range and a preset scale range, where the angle range is denoted as... The scaling range is denoted as The rotation step size is Scaling step size is At each level of the pyramid, with the geometric center of the template image as the transformation origin, all combinations of angle and scale values ​​are iterated one by one according to the angle sequence and scale sequence to generate a template set covering the entire angle search range and scale search range.

[0047] Step S403: Match each template in the template set with the target image; calculate the average value of the dot product of unit direction vectors as the similarity.

[0048] One-to-one matching, also known as layer-by-layer search, refers to starting the matching from the top of the pyramid, using the matching results of the previous layer as the reference position for the search of the next layer, and refining it layer by layer until the bottom layer.

[0049] Specifically, during matching, the average of the dot products of the unit direction vectors of all points in the template and the corresponding unit direction vectors in the target image is calculated as the similarity score. The similarity formula is as follows: ; in, For template number The unit direction vector of each feature point Let be the unit direction vector at the corresponding point in the target image. N The number of template feature points is denoted by , and the similarity score ranges from -1 to 1. A higher value indicates a higher degree of matching.

[0050] Step S404: Take the position and angle with the highest similarity as the best matching result for this layer.

[0051] The best matching result includes the best matching position and the best matching angle of the layer, which represents the optimal pose estimate of the template in the target image at the current resolution scale.

[0052] Step S405: Is the current level the lowest level?

[0053] Step S405 is a judgment step. If the judgment is yes, that is, the current level is the bottom level, then step S406 is executed; if the judgment is no, that is, the current level is not the bottom level, then step S407 is executed.

[0054] Step S406: Output the initial position and initial angle.

[0055] The matching process begins at the top of the pyramid. Each template in the template set is compared to the target image to calculate its similarity, and the position with the highest similarity is taken as the optimal position for that layer. When moving to the next layer, the optimal position from the previous layer is mapped to the corresponding coordinates of the current layer as the search center, and matching continues within that neighborhood. This process continues until the bottom layer matching is complete, outputting the initial pixel-level positions. and initial angle At the same time, the initial scaling factor is obtained. This is used for initialization in the subsequent fine matching stage.

[0056] Step S407: Map the best position in this layer to the next layer as the search center for the next layer and continue matching.

[0057] Mapping refers to converting the optimal position coordinates of the previous layer to the corresponding coordinates of the next layer according to the resolution ratio. Then, in the next layer, matching searches continue within the local neighborhood centered on that coordinate position. By progressively passing the search center layer by layer, the search scope gradually narrows from a global search at the top level to a fine-grained local search at the bottom level, achieving a layered matching strategy from coarse to fine. The optimal position refers to the position with the highest similarity in the current layer.

[0058] As can be seen, in this example, by constructing an image pyramid for coarse-to-fine layered matching, a large number of non-target regions are quickly eliminated at the low-resolution top layer, and then the high-resolution bottom layer is gradually and accurately located, which greatly reduces the computational load of coarse matching and improves the matching speed. At the same time, by generating a template set covering the preset angle range and scale range, the initial matching result is made adaptable to the rotation and scale changes of the target, providing an initial pose close to the true value for the nonlinear optimization in the subsequent fine matching stage, which helps to ensure the convergence speed of fine matching and the final positioning accuracy.

[0059] In one possible embodiment, at each layer of the image pyramid, a template set with different angles and scales is generated for the template image, including: rotating the template image according to a preset angle search range and angle step size to generate a rotated template image; scaling the template image according to a preset scale search range to generate a scaled template image; and generating a template set based on the rotated template image and the scaled template image.

[0060] Rotation transformation refers to rotating the template image around its geometric center at a preset angle within the image plane. After rotation, each pixel in the template image experiences a coordinate shift relative to its original position. Scaling transformation refers to scaling the template image by a preset scale factor, making it proportionally enlarged or reduced in both the horizontal and vertical directions. After scaling, each pixel in the template image experiences coordinate scaling relative to its original position. After rotation and scaling transformations, the spatial distribution of the template image's pixels changes. Therefore, interpolation methods (such as bilinear interpolation or nearest neighbor interpolation) are used to assign grayscale values ​​to the non-integer coordinates after the transformation to ensure the integrity and continuity of the transformed template image.

[0061] Within the current pyramid level, the search range is determined based on a preset angle. Compared to the angle step size (the aforementioned rotation step size). Generate a discretized angle sequence: ; in, =0, 1, ..., , ; Search range based on preset scale With scaling step Generate a discretized scale sequence: ; in, =0, 1, ..., , .

[0062] Furthermore, for each set of angle values and scale value Each combination applies a scaling and rotation transformation to the template image sequentially, generating a corresponding transformed template instance. The transformation origin is the geometric center of the template image, and the transformation order is scaling followed by rotation. This process is repeated for all combinations to generate a final template instance containing... A set of template instances, used for subsequent matching with the target image.

[0063] As can be seen, in this example, by setting the angle search range and scale search range and combining them with the discretization step size to generate a discrete sequence, and performing traversal rotation and scaling transformations on the template image, the generated template set can comprehensively cover various angle and scale changes that the target may have. This ensures that the initial matching result closest to the target can be captured under different pose conditions during the coarse matching stage, thereby improving the recall and robustness of coarse matching. At the same time, the size of the template set can be controlled by flexibly adjusting the angle step size and scaling step size, achieving a balance between matching accuracy and computational efficiency.

[0064] Step S204: Based on the current matching pose, which is composed of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset, perform coordinate transformation on the template feature point set.

[0065] The initial values ​​for position offset and angle offset are zero. Position offset refers to the amount of fine-tuning to be optimized in the X and Y directions of the template feature point relative to the initial position in the target image coordinate system, and angle offset refers to the amount of fine-tuning to be optimized in the template feature point relative to the initial angle. Position offset and angle offset together constitute the optimization variables; coordinate transformation refers to the spatial geometric transformation that maps the template feature point from the template coordinate system to the target image coordinate system according to the current matching pose.

[0066] The design principle of this step is to model the template matching problem as a parameter optimization problem. By continuously adjusting the position offset and angle offset, the current matching pose is updated so that the template feature points gradually approach the real edge position in the target image.

[0067] Step S205: Obtain the gradient intensity values ​​at multiple mapping points after coordinate transformation in the gradient intensity map using interpolation.

[0068] Here, the mapping point refers to the sub-pixel level coordinate position of the template feature point in the target image coordinate system after coordinate transformation; the interpolation method refers to the numerical calculation method that estimates the sub-pixel precision gradient intensity value at the mapping point by using the gradient intensity values ​​at integer pixel positions around the mapping point through weighted calculation.

[0069] The design principle of this step is that the coordinates of the mapped points after coordinate transformation are usually not integer pixel positions, and cannot be directly indexed from the gradient intensity map. The interpolation method can estimate the gradient intensity value at any sub-pixel position with continuous accuracy, thereby providing accurate objective function values ​​for subsequent nonlinear optimization.

[0070] In one possible embodiment, after obtaining the gradient intensity values ​​at multiple mapping points after coordinate transformation in the gradient intensity map by interpolation in step S205, the method further includes a step of effectively filtering the mapping points: based on the comparison results between the gradient intensity values ​​at each mapping point and the preset filtering threshold, effective feature points are selected from the multiple mapping points, and only the gradient intensity values ​​corresponding to the effective feature points are used in the calculation of the objective function value.

[0071] Among them, effective feature points refer to mapping points whose gradient intensity values ​​are greater than the preset screening threshold. Their corresponding target image locations are in areas with significant edges, which can provide effective guidance information for optimization. Mapping points whose gradient intensity values ​​are less than or equal to the preset screening threshold correspond to target image locations in flat areas or weak edge areas, which have a weaker guiding effect on optimization or may even cause interference.

[0072] Furthermore, as the optimization iterations proceed, the current matching pose is continuously updated, and the positions of each mapping point on the gradient intensity map change accordingly, as do their gradient intensity values. Therefore, the set of effective feature points may differ in different iteration steps. Specifically, after completing the coordinate transformation and interpolation of the mapping points in each iteration, the gradient intensity values ​​of all mapping points are compared with a preset screening threshold. When the gradient intensity value of a mapping point is greater than the preset screening threshold, the mapping point is marked as a valid feature point, and its gradient intensity value participates in the accumulation of the objective function value in the current iteration step; when the gradient intensity value of a mapping point is less than or equal to the preset screening threshold, the mapping point is marked as an invalid feature point, and its gradient intensity value does not participate in the calculation of the objective function value in the current iteration step.

[0073] Furthermore, when the proportion of invalid feature points exceeds a preset threshold, it indicates that a large number of feature points in the current template feature point set fall within flat areas of the target image, resulting in poor matching quality between the template feature point set and the current region of the target image. In this case, a feature point resampling operation is triggered: a preset number of edge points are selected in the template image in descending order of gradient intensity as new feature points, replacing the feature points with the lowest gradient intensity contribution in the current template feature point set, thus updating the template feature point set participating in subsequent iterations. If the proportion of valid feature points is lower than a preset threshold, the feature points with the lowest contribution in the current template feature point set are removed from the feature point set, and the same number of new feature points are resampled from the edge points of the template image to supplement them, ensuring that the total number of feature points participating in subsequent iterations remains unchanged. The resampling criteria include the gradient intensity ranking position of the new feature points in the template image and the spatial uniformity of the new feature points in the template image.

[0074] As can be seen in this example, through the above-mentioned filtering and resampling mechanism, the feature point set participating in the optimization in subsequent iterations can better adapt to the current gradient intensity distribution of the target image, effectively reduce the interference of flat areas and weak edge areas on the optimization direction, and further enhance the algorithm's adaptability and positioning stability in different target image regions and different iteration stages.

[0075] In one possible embodiment, the gradient intensity values ​​at multiple mapped points after coordinate transformation are obtained in the gradient intensity map by interpolation, including: determining the pixels in the neighborhood around each mapped point; and calculating the gradient intensity value at each mapped point based on the pixels in the neighborhood around each mapped point using any one of bilinear interpolation, cubic interpolation, or B-spline interpolation.

[0076] Here, the mapping point refers to the sub-pixel level coordinate position of the template feature point after coordinate transformation and mapped to the target image coordinate system; the pixels in the surrounding neighborhood refer to the pixels in the gradient intensity map that are spatially closest to the mapping point; the interpolation method is a numerical method that estimates the sub-pixel precision gradient intensity value at the mapping point by using the gradient intensity values ​​at the known integer pixel positions in the surrounding neighborhood of the mapping point through weighted calculation.

[0077] Taking bilinear interpolation as an example, let the sub-pixel level coordinates of the mapping point be... Take its integer part as The decimal part is The calculation formula is as follows: ; in, and for The decimal part within the range. The four pixels in the neighborhood surrounding the mapped point are respectively , , , The corresponding gradient intensity values ​​are respectively , , , The gradient intensity at the mapping point is then calculated using the bilinear interpolation formula: .

[0078] Alternatively, provided that the accuracy requirements are met, cubic interpolation or B-spline interpolation can be used instead of bilinear interpolation. Cubic interpolation uses sixteen pixels in a 4×4 neighborhood around the mapped point to perform cubic polynomial weighted calculation, which has higher accuracy but also increases the amount of computation accordingly. B-spline interpolation uses B-spline basis functions to perform smooth weighting on the pixels in the surrounding neighborhood, which has good smoothness and local support.

[0079] As can be seen, in this example, the sub-pixel-level gradient intensity value at the mapping point is obtained on the gradient intensity map by interpolation, so that small changes in position and angle offset can be continuously responded to in the objective function value, providing accurate gradient intensity information for nonlinear optimization. At the same time, bilinear interpolation is simple to implement and has high computational efficiency, which is conducive to maintaining a high processing speed in industrial settings. Cubic interpolation and B-spline interpolation can be flexibly selected as alternatives in scenarios with higher accuracy requirements.

[0080] In one possible embodiment, after obtaining the gradient intensity values ​​at multiple mapping points after coordinate transformation in the gradient intensity map by interpolation, the method further includes: selecting effective feature points from the multiple mapping points based on the comparison results of the gradient intensity values ​​at each mapping point and a preset screening threshold, and using only the gradient intensity values ​​corresponding to the effective feature points in the calculation of the objective function value.

[0081] In this context, effective feature points refer to mapping points whose gradient intensity values ​​are greater than a preset screening threshold. Their corresponding target image locations are in areas with significant edges, providing effective guidance for optimization. Mapping points whose gradient intensity values ​​are less than or equal to the preset screening threshold correspond to target image locations in flat or weakly edged areas, offering weaker guidance for optimization and potentially causing interference. As the optimization iteration progresses, the current matching pose is continuously updated, and the positions of each mapping point on the gradient intensity map change accordingly, as do their gradient intensity values. Therefore, the set of effective feature points may differ in different iteration steps.

[0082] Specifically, after completing the coordinate transformation and interpolation of the mapped points in each iteration, the gradient intensity values ​​of all mapped points are compared with a preset screening threshold. When the gradient intensity value of a mapped point is greater than the preset screening threshold, the mapped point is marked as a valid feature point, and its gradient intensity value participates in the accumulation of the objective function value in the current iteration step; when the gradient intensity value of a mapped point is less than or equal to the preset screening threshold, the mapped point is marked as an invalid feature point, and its gradient intensity value does not participate in the calculation of the objective function value in the current iteration step. Through the above screening mechanism, only feature points located in the significant edge regions of the target image are retained for optimization.

[0083] Furthermore, when the proportion of invalid feature points exceeds a preset threshold, it indicates that a large number of feature points in the current template feature point set fall into flat areas of the target image, resulting in poor matching quality between the template feature point set and the current region of the target image. In this case, a feature point resampling operation is triggered: a preset number of edge points are selected in the template image in descending order of gradient intensity as new feature points, replacing the feature points with the lowest gradient intensity contribution in the current template feature point set, thus updating the template feature point set participating in subsequent iterations. If the proportion of valid feature points is lower than the preset threshold, the feature points with the lowest contribution in the current template feature point set are removed from the feature point set, and the same number of new feature points are resampled from the edge points of the template image to supplement them, ensuring that the total number of feature points participating in subsequent iterations remains unchanged. The resampling criteria include the gradient intensity ranking position of the new feature points in the template image and the spatial uniformity of the new feature points in the template image. Through the above filtering and resampling, the feature point set participating in subsequent iterations can better adapt to the current gradient intensity distribution of the target image.

[0084] As can be seen, in this embodiment, by filtering the gradient intensity values ​​of the mapping points in each iteration and marking low-quality feature points as invalid, only retaining valid feature points to participate in the optimization, the interference of flat areas and weak edge areas on the optimization direction is effectively reduced. At the same time, by dynamically monitoring the proportion of valid feature points and triggering resampling when the proportion is too low, the feature point set can be adaptively updated during the iteration process, further enhancing the algorithm's adaptability and positioning stability in different target image regions and different iteration stages.

[0085] Step S206: Using position offset and angle offset as optimization variables and maximizing the sum of gradient intensity values ​​as the optimization objective, iteratively update the position offset and angle offset until the convergence condition is met, and output the sub-pixel level target position and target angle.

[0086] Among them, the gradient intensity sum refers to the cumulative sum of the gradient intensity values ​​of all transformed template feature points at their corresponding positions on the gradient intensity map of the target image, which is used to measure the degree of matching between the template and the edge of the target image under the current matching pose; the convergence condition refers to the pre-set judgment criteria for terminating the iteration, which is used to ensure that the optimization process stops after reaching sufficient accuracy; sub-pixel level refers to the positioning accuracy reaching a higher accuracy below the pixel level, that is, the error is less than one pixel.

[0087] For example, the above iterative update can be solved using a derivative-free optimization method. In other embodiments, other optimization algorithms can be used instead of the above method, including but not limited to gradient descent, Newton's method, or the Levenberg-Marquardt Algorithm (LM algorithm). All of these alternative algorithms can be used to iteratively update the position and angle offsets to achieve the optimization objective of maximizing the sum of gradient intensity values.

[0088] The design principle of this step is that when the template feature points completely coincide with the real target edge in the target image, the gradient intensity value of the location of these feature points reaches a local maximum. Therefore, by maximizing the sum of the gradient intensity values ​​of all feature points, the optimization iteration is driven, so that the matching pose continuously approaches the real value, and finally sub-pixel level high-precision positioning is achieved.

[0089] To better illustrate the implementation details of the fine matching process below, please refer to the following examples. Figure 5 , Figure 5 This is a schematic diagram of a fine-matching process provided in an embodiment of this application; as shown... Figure 5 As shown, the fine matching process includes: Step S501, Input: coarse matching result, gradient intensity map, template feature point set; initialize position offset and angle offset to zero; construct a simplex with the initial point as the center.

[0090] Step S502: Perform coordinate transformation on the template feature point set according to the current matching pose; the current matching pose consists of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset.

[0091] Step S503: Obtain the gradient intensity value at the mapping point in the gradient intensity map by interpolation.

[0092] Step S504: Calculate the sum of gradient intensity values ​​corresponding to the current vertex as the objective function value.

[0093] Step S505: Sort the simplex vertices according to the objective function values ​​of each vertex, and update them through reflection, expansion, contraction, or shrinkage operations.

[0094] Step S506: Does the convergence condition meet?

[0095] The convergence conditions include: the standard deviation of the objective function value corresponding to each vertex of the simplex is less than the preset convergence threshold, or the number of iterations reaches the maximum number of iterations. Step S506 is a judgment step. If the judgment is yes, that is, the convergence condition is met, then step S507 is executed; if the judgment is no, that is, the convergence condition is not met, then step S508 is executed.

[0096] Step S507: Output the optimal position offset and the optimal angle offset, and calculate the sub-pixel level target position and target angle based on the sum of the initial position and the optimal position offset, and the sum of the initial angle and the optimal angle offset.

[0097] Step S508: Return to step S502 and continue the next round of iterative optimization until the convergence condition is met.

[0098] The simplex construction method, the specific calculation method of the objective function value of each vertex, the detailed execution logic of each geometric operation such as reflection / expansion / contraction / shrinkage, the specific setting of the convergence threshold, and the determination method of iteration termination involved in the above-mentioned fine matching process will be explained in detail in subsequent embodiments with specific formulas and parameters.

[0099] In one possible embodiment, position offset and angle offset are used as optimization variables, and maximizing the sum of gradient intensity values ​​is used as the optimization objective. The position offset and angle offset are iteratively updated, including: constructing an optimization space with position offset and angle offset as optimization variables, and constructing a simplex with the initial value as the center. Each vertex of the simplex is used to calculate the corresponding sum of gradient intensity values ​​in each iteration; sorting according to the sum of gradient intensity values ​​corresponding to each vertex; if the standard deviation of the sum of gradient intensity values ​​corresponding to each vertex of the simplex is less than a preset convergence threshold or the maximum number of iterations is reached, the iteration is stopped.

[0100] The preset convergence threshold and maximum number of iterations are used as the criteria for determining convergence. A simplex is a polyhedron in three-dimensional optimization space consisting of four vertices, each vertex corresponding to a set of positional and angular offset values. Optimization space is a three-dimensional space constructed using positional and angular offsets as coordinate axes, with the three dimensions being positional offset in the X direction, positional offset in the Y direction, and angular offset. The sum of gradient intensity values ​​refers to the cumulative sum of gradient intensity values ​​obtained by interpolation on the gradient intensity map for all template feature points at a given pose of a vertex of the simplex after coordinate transformation. The standard deviation measures the dispersion of the objective function values ​​among the vertices of the simplex; a smaller standard deviation indicates that the objective function values ​​of each vertex are closer, and the optimization process tends to be stable.

[0101] Among them, the initial point Construct an initial simplex around the center, which contains four vertices: , , , ,in As the initial vertex, , , These are the vertices offset by an initial step size relative to the initial vertex in the X-direction, Y-direction, and angular offset dimensions, respectively. In each iteration, for each vertex of the simplex... = Calculate the objective function value corresponding to this vertex. The calculation formula is: ; in, To optimize the variables, let X represent the position offset in the X direction, the position offset in the Y direction, and the angular offset, respectively; This represents the total number of feature points in the template. For the first Each template feature point, after coordinate transformation based on its current pose, has sub-pixel-level coordinates in the target image. The current pose is represented as follows: ; This represents the gradient intensity value at this mapping point, obtained through interpolation on the gradient intensity map. This is based on the objective function values ​​of all vertices. Sort the vertices from largest to smallest objective function value. The vertex with the largest objective function value is the optimal point, and the vertex with the smallest objective function value is the worst point. The convergence criterion is: calculate the standard deviation of the objective function values ​​of all vertices of the simplex. ; in, The total number of vertices of the simplex. =4; For the first One vertex; For the first The objective function values ​​corresponding to each vertex; This is the average of the objective function values ​​for all vertices. If... Or the number of iterations reaches If the iteration stops, then the iteration stops. To preset the convergence threshold, This represents the maximum number of iterations.

[0102] As can be seen, in this example, by constructing a simplex in the three-dimensional optimization space and sorting and judging the sum of the gradient strength values ​​of each vertex, multivariate joint optimization can be achieved without needing to solve the gradient of the objective function, thus avoiding the inapplicability of gradient calculation to non-smooth objective functions. At the same time, using the standard deviation of the objective function value at the vertex of the simplex as the convergence criterion can effectively reflect whether the optimization process tends to be stable, avoiding the waste of computational resources due to invalid iterations.

[0103] In one possible embodiment, after sorting according to the sum of gradient intensity values ​​corresponding to each vertex, the method further includes: determining the optimal point, the second worst point, and the worst point based on the sorting result; calculating the centroids of the vertices in the simplex except for the worst point; calculating the reflection point of the worst point relative to the centroid, and calculating the sum of gradient intensity values ​​of the reflection points; if the sum of gradient intensity values ​​of the reflection points is greater than the sum of gradient intensity values ​​of the optimal point, then calculating an extension point, and replacing the worst point with the extension point or the point with the larger sum of gradient intensity values ​​among the reflection points; if the sum of gradient intensity values ​​of the reflection points is between the sum of gradient intensity values ​​of the second worst point and the sum of gradient intensity values ​​of the optimal point, then replacing the worst point with the reflection point; if the sum of gradient intensity values ​​of the reflection points is greater than the sum of gradient intensity values ​​of the worst point and less than or equal to the sum of gradient intensity values ​​of the second worst point, then calculating an outer contraction point, and replacing the worst point with the outer contraction point; if the sum of gradient intensity values ​​of the reflection points is less than or equal to the sum of gradient intensity values ​​of the worst point, then calculating an inner contraction point, and replacing the worst point with the inner contraction point.

[0104] The optimal point is the vertex in the current simplex with the largest objective function value (i.e., the largest sum of gradient strength), corresponding to the best set of position and angle offsets. The worst point is the vertex in the current simplex with the smallest objective function value (i.e., the smallest sum of gradient strength), corresponding to the worst set of position and angle offsets. The second worst point is the vertex in the current simplex with an objective function value between the optimal and worst points (i.e., the sum of gradient strength between the optimal and worst points). The centroid is the geometric center of the three vertices other than the worst point. The reflection point is the point symmetrical to the worst point about the centroid. The extension point is the point obtained by further extending in the reflection direction. The external contraction point is the point obtained by contracting in the reflection direction. The internal contraction point is the point obtained by contracting in the opposite direction of the line connecting the worst point and the centroid.

[0105] First, calculate the value of the simplex excluding the worst point. The other three vertices , , Center of gravity: ; in, For the optimal point, For the second-lowest point, It is the second worst point (i.e., the other vertex besides the worst point and the second worst point). The center of gravity for these three factors is then determined. The worst-case scenario is then calculated. Regarding the center of gravity Reflection point: ; in, The reflection coefficient, =1.0, This is the reflection point.

[0106] Based on the reflection point objective function value Based on the comparison with the current vertex objective function value, perform the following branch operation: Case 1 (Extended): If (If the reflection point is better than the optimal point), then calculate the extension point: ; in, For expansion coefficient, =2.0, This is the expansion point. If Then replace Otherwise, with replace .

[0107] Case 2 (Reflection Replacement): If (If the reflection point is better than or equal to the best point but worse than or equal to the second worst point), then... replace .

[0108] Case 3 (External contraction): If (If the reflection point is better than the second worst point but worse than the second worst point), then calculate the external contraction point: ; in, The shrinkage coefficient, =0.5, This is the point of external contraction. If... Then replace Otherwise, perform a shrink operation.

[0109] Case 4 (Internal Contraction): If (If the reflection point is worse than the second worst point), then calculate the inward contraction point: ; in, The shrinkage coefficient, =0.5, This is the point of internal contraction. If... Then replace Otherwise, perform a shrink operation: ; in, To reduce the factor, =0.5, To remove The remaining vertices, i.e. all non-optimal vertices, shrink towards the optimal point.

[0110] As can be seen in this example, through five geometric operations—reflection, expansion, external contraction, internal contraction, and shrinkage—the simplex can adaptively move, expand, and shrink in the optimization space. It can perform global exploration over a large area to escape local optima, and perform fine-grained search near the optimal solution to improve positioning accuracy. At the same time, this process does not require calculating the gradient information of the objective function and is still applicable to non-smooth objective functions introduced by interpolation, demonstrating good robustness and practicality.

[0111] In one possible embodiment, before updating the simplex vertices through reflection, expansion, contraction, or shrinkage operations, the method further includes: acquiring historical iteration data of the current simplex in the most recent iterations; determining the current optimization state based on the historical iteration data; and adjusting the initial step size or operation mode of the simplex based on the current optimization state.

[0112] Among them, historical iteration data refers to the record of changes in the position and angle offsets of the optimal vertex of the simplex in the preset number of iterations before the current iteration step, which is used to reflect the progress speed and trend of the optimization process in a recent period of time; the current optimization state is the classification of the current stage of the optimization process, including fast convergence state, slow convergence state, and stagnation state; the initial step size refers to the offset in each dimension when constructing the initial simplex; the operation mode refers to the coefficient combination method used for geometric operations such as reflection, expansion, contraction, and shrinkage, which can be adjusted by adjusting the step size or operation mode to adapt to different optimization stages.

[0113] Specifically, the pose change of the optimal vertex of the simplex is recorded in the most recent iterations. If the pose change remains large and shows a continuous decreasing trend in multiple iterations, it is determined that the current state is fast convergence. In this case, the current simplex step size is kept unchanged or the step size is increased appropriately to maintain a fast search speed. If the pose change gradually decreases to a small value in multiple iterations, it is determined that the current state is slow convergence. In this case, the simplex step size is reduced appropriately to narrow the search range to a smaller neighborhood near the current optimal solution for fine-grained search to improve output accuracy. If the pose change is less than the preset stagnation threshold in multiple iterations, it is determined that the current state is stagnant. In this case, the simplex step size is actively increased and an expansion operation is triggered to allow the simplex to explore a wider region in the optimization space in an attempt to escape possible local flat regions. When the simplex is determined to be in a slow convergence state, the initial step size of the simplex is gradually reduced. When the simplex is determined to be in a stagnant state, an expansion search operation is triggered, temporarily increasing the step size of each vertex of the simplex to a multiple of the original step size, and exploring a large area around the current optimal solution. If the objective function value is significantly improved after the expansion search, the expanded step size is maintained and iteration continues. If the objective function value is not significantly improved after the expansion search, the original step size is restored and the fine convergence mode is entered, and the current optimal solution is finely approximated by the reduction operation.

[0114] As can be seen in this example, by monitoring historical data during the iteration process to dynamically determine the optimization state, and adaptively adjusting the simplex step size and operation mode according to different states, it can maintain a relatively fast convergence speed in the early stage of optimization, perform fine search to improve positioning accuracy in the later stage of optimization, and actively expand the search range to jump out of the local area when stagnant, further improving the convergence efficiency and final positioning accuracy in the fine matching stage.

[0115] As can be seen, in this embodiment, by performing edge detection and non-maximum suppression filtering on the template image and the target image, weak edges and noise points are effectively removed, the size of the feature point set is compressed, and the reliability of the initial pose of the coarse matching is ensured. By constructing an image pyramid for hierarchical matching from coarse to fine, the computational load of coarse matching is significantly reduced. At the same time, by generating a template set covering the preset angle search range and scale search range, the adaptability of the initial matching result to target rotation and scale changes is ensured. In the fine matching stage, the sub-pixel-level gradient intensity value at the mapping point is obtained on the gradient intensity map by interpolation, so that small changes in position and angle offset can be continuously responded to on the objective function, providing accurate optimization information for nonlinear optimization. By constructing a simplex in the three-dimensional optimization space and performing iterative operations such as reflection, expansion, contraction, and shrinkage based on a preset algorithm, adaptive optimization solution without gradient information is achieved, effectively overcoming the dependence of traditional surface fitting methods on local smoothness and unimodality. At the same time, multi-dimensional joint optimization avoids the accumulation of step-by-step errors, significantly improving the positioning accuracy and noise robustness of template matching, and meeting the needs of high-precision visual positioning in complex industrial scenarios.

[0116] The following are embodiments of the apparatus of this application. These embodiments of the apparatus and the embodiments of the method of this application belong to the same concept and are used to execute the methods described in the embodiments of this application. For ease of explanation, only the parts related to the apparatus embodiments of this application are shown in the embodiments of this application. For specific technical details not disclosed, please refer to the description of the embodiments of the method of this application, which will not be repeated here.

[0117] This application provides a subpixel template matching device based on nonlinear optimization, used to execute the steps in the above-described subpixel template matching method based on nonlinear optimization. The subpixel template matching device based on nonlinear optimization in this application may include modules corresponding to the respective steps.

[0118] This application embodiment can divide the sub-pixel template matching device based on nonlinear optimization into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. The module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0119] When dividing each function into modules according to its corresponding function. Figure 6 This is a functional unit block diagram of a sub-pixel template matching device based on nonlinear optimization provided in an embodiment of this application; the sub-pixel template matching device based on nonlinear optimization is applied to... Figure 1The visual processing server 120 in the visual positioning system 10 shown in Figure 6. As shown in Figure 6, the sub-pixel template matching device 60 based on nonlinear optimization includes an image acquisition unit 601 for acquiring a template image and a target image; a feature gradient generation unit 602 for extracting edge feature points from the template image and the target image respectively, obtaining a template feature point set and a target feature point set, and constructing a gradient intensity map, which characterizes the gradient intensity at each pixel position in the target image; an initial matching unit 603 for performing coarse matching based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image; and a pose transformation unit 604. The first unit is used to perform coordinate transformation on the template feature point set based on the current matching pose consisting of the sum of the initial position and position offset, and the sum of the initial angle and angle offset. The initial values ​​of the position offset and angle offset are zero. The second unit is used to obtain the gradient intensity values ​​at multiple mapped points after coordinate transformation in the gradient intensity map through interpolation. The third unit is used to optimize the output unit 606, which uses the position offset and angle offset as optimization variables and maximizes the sum of gradient intensity values ​​as the optimization objective, and iteratively updates the position offset and angle offset until the convergence condition is met, and outputs the sub-pixel level target position and target angle.

[0120] In one possible embodiment, in extracting edge feature points from the template image and the target image respectively to obtain a template feature point set and a target feature point set, the feature gradient generation unit 602 is specifically used to: perform edge detection on the template image and the target image respectively to obtain a template edge point set and a target edge point set respectively; calculate the gradient intensity of each edge point in the template edge point set and the target edge point set respectively; perform non-maximum suppression on the edge points in the template edge point set and the target edge point set respectively, and select edge points whose gradient intensity is a local maximum in a preset neighborhood as the template feature point set and the target feature point set.

[0121] In one possible embodiment, in determining the initial position and initial angle of the template image in the target image by performing coarse matching based on the template feature point set and the target feature point set, the initial matching unit 603 is specifically used to: construct image pyramids for the template image and the target image respectively; generate template sets with different angles and scales for the template image at each layer of the image pyramid; match the templates in the template set with the target image one by one, and perform a layer-by-layer search from the top to the bottom of the image pyramid, using the best position obtained from each layer as the search center for the next layer, to obtain the initial position and initial angle.

[0122] In one possible embodiment, in generating template sets with different angles and scales for the template image at each layer of the image pyramid, the initial matching unit 603 is specifically used to: rotate the template image according to a preset angle search range and angle step size to generate a rotated template image; scale the template image according to a preset scale search range to generate a scaled template image; and generate a template set based on the rotated template image and the scaled template image.

[0123] In one possible embodiment, in obtaining gradient intensity values ​​at multiple mapped points after coordinate transformation in the gradient intensity map by interpolation, the gradient interpolation unit 605 is specifically used to: determine the pixels in the neighborhood around each mapped point; and calculate the gradient intensity value at each mapped point based on the pixels in the neighborhood around each mapped point using any one of bilinear interpolation, cubic interpolation, or B-spline interpolation.

[0124] In one possible embodiment, in iteratively updating the position and angle offsets with position and angle offsets as optimization variables and maximizing the sum of gradient intensity values ​​as the optimization objective, the optimization output unit 606 is specifically used to: construct an optimization space with position and angle offsets as optimization variables, and construct a simplex centered on the initial value, where each vertex of the simplex is used to calculate the corresponding sum of gradient intensity values ​​in each iteration; sort the values ​​according to the sum of gradient intensity values ​​corresponding to each vertex; and stop iterating if the standard deviation of the sum of gradient intensity values ​​corresponding to each vertex of the simplex is less than a preset convergence threshold or the maximum number of iterations is reached, wherein the preset convergence threshold and the maximum number of iterations are the criteria for determining the convergence condition.

[0125] In one possible embodiment, after sorting according to the sum of gradient intensity values ​​corresponding to each vertex, the optimization output unit 606 is further configured to: determine the optimal point, the second worst point, and the worst point based on the sorting result, wherein the optimal point has the largest sum of gradient intensity values, the worst point has the smallest sum of gradient intensity values, and the second worst point has a sum of gradient intensity values ​​between the optimal point and the worst point; calculate the centroids of the vertices in the simplex except for the worst point; calculate the reflection point of the worst point relative to the centroid, and calculate the sum of gradient intensity values ​​of the reflection point; if the sum of gradient intensity values ​​of the reflection point is greater than the sum of gradient intensity values ​​of the optimal point, Then, calculate the expansion point and replace the worst point with the point that has the larger sum of gradient intensity values ​​among the expansion points or reflection points; if the sum of gradient intensity values ​​of the reflection points is between the sum of gradient intensity values ​​of the second worst point and the sum of gradient intensity values ​​of the best point, then replace the worst point with the reflection point; if the sum of gradient intensity values ​​of the reflection points is greater than the sum of gradient intensity values ​​of the worst point and less than or equal to the sum of gradient intensity values ​​of the second worst point, then calculate the outer contraction point and replace the worst point with the outer contraction point; if the sum of gradient intensity values ​​of the reflection points is less than or equal to the sum of gradient intensity values ​​of the worst point, then calculate the inner contraction point and replace the worst point with the inner contraction point.

[0126] When using integrated units, such as Figure 7 As shown, Figure 7 This is a functional unit block diagram of another sub-pixel template matching device based on nonlinear optimization provided in an embodiment of this application. Figure 7 The subpixel template matching device 60 based on nonlinear optimization includes a processing module 702 and a communication module 701. The processing module 702 controls and manages the actions of the subpixel template matching device 60, including, for example, the steps of the image acquisition unit 601, feature gradient generation unit 602, initial matching unit 603, pose transformation unit 604, gradient interpolation unit 605, and optimization output unit 606, and / or other processes for executing the techniques described herein. The communication module 701 supports interaction between the subpixel template matching device and other devices. Figure 7 As shown, the subpixel template matching device based on nonlinear optimization may include a storage module 703, which is used to store the program code and data of the subpixel template matching device based on nonlinear optimization.

[0127] The processing module 702 can be a processor or processing module, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 701 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 703 can be a memory.

[0128] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The above-mentioned sub-pixel template matching device 60 based on nonlinear optimization can all perform the above-mentioned... Figure 2 The subpixel template matching method shown is based on nonlinear optimization.

[0129] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0130] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. For example... Figure 8As shown, the electronic device 80 may include one or more of the following components: a processor 801 and a memory 802 coupled to the processor 801, wherein the memory 802 may store one or more computer programs 803, and the one or more computer programs 803 may be configured to implement the methods described in the above embodiments when executed by one or more processors 801. The electronic device 80 here is the vision processing server 120 in the above embodiments.

[0131] Processor 801 may include one or more processing cores. Processor 801 connects to various parts within the electronic device 80 using various interfaces and lines, and performs various functions and processes data of the electronic device 80 by running or executing instructions, programs, code sets, or instruction sets stored in memory 802, and by calling data stored in memory 802. Optionally, processor 801 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 801 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 801 and may be implemented separately using a communication chip.

[0132] The memory 802 may include random access memory (RAM) or read-only memory (ROM). The memory 802 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 802 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the electronic device 80 during use.

[0133] It is understood that the electronic device 80 may include more or fewer structural elements than those shown in the above block diagram, without limitation herein.

[0134] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.

[0135] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0136] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0140] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0141] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A sub-pixel template matching method based on nonlinear optimization, characterized in that, include: Obtain the template image and the target image; Edge feature points are extracted from the template image and the target image respectively to obtain a template feature point set and a target feature point set, and a gradient intensity map is constructed. The gradient intensity map is used to characterize the gradient intensity at each pixel position of the target image. Based on the template feature point set and the target feature point set, a coarse matching is performed to determine the initial position and initial angle of the template image in the target image; Based on the current matching pose, which is composed of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset, the template feature point set is subjected to coordinate transformation, and the initial values ​​of the position offset and the angle offset are zero. The gradient intensity values ​​at multiple mapping points after the coordinate transformation are obtained in the gradient intensity map by interpolation. Using the position offset and the angle offset as optimization variables, and maximizing the sum of the gradient intensity values ​​as the optimization objective, the position offset and the angle offset are iteratively updated until the convergence condition is met, and the sub-pixel level target position and target angle are output. The iterative update of the position offset and the angle offset includes: constructing an optimization space with the position offset and the angle offset as optimization variables, and constructing a simplex with the initial value as the center, wherein each vertex of the simplex is used to calculate the sum of the corresponding gradient intensity values ​​in each iteration; Sort according to the sum of the gradient intensity values ​​corresponding to each vertex; If the standard deviation of the sum of gradient intensity values ​​corresponding to each vertex of the simplex is less than a preset convergence threshold or the maximum number of iterations is reached, then the iteration stops. The preset convergence threshold and the maximum number of iterations are the criteria for determining the convergence condition.

2. The method according to claim 1, characterized in that, The step of extracting edge feature points from the template image and the target image respectively to obtain a template feature point set and a target feature point set includes: Edge detection is performed on the template image and the target image respectively to obtain the template edge point set and the target edge point set respectively; Calculate the gradient intensity of each edge point in the template edge point set and the target edge point set, respectively; Non-maximum suppression is performed on the edge points of the template edge point set and the target edge point set respectively, and edge points whose gradient intensity is a local maximum in a preset neighborhood are selected as the template feature point set and the target feature point set.

3. The method according to claim 2, characterized in that, The step of performing coarse matching based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image includes: Construct image pyramids for the template image and the target image respectively; At each level of the image pyramid, a set of templates with different angles and scales is generated for the template image; The templates in the template set are matched one by one with the target image. From the top to the bottom of the image pyramid, the best position obtained from each layer is used as the search center for the next layer to search layer by layer, so as to obtain the initial position and the initial angle.

4. The method according to claim 3, characterized in that, The process of generating template sets with different angles and scales for the template image at each layer of the image pyramid includes: The template image is rotated according to the preset angle search range and angle step size to generate the rotated template image; The template image is scaled and transformed according to a preset scale search range to generate the scaled and transformed template image. The template set is generated based on the rotated template image and the scaled template image.

5. The method according to any one of claims 1-4, characterized in that, The step of obtaining the gradient intensity values ​​at multiple mapped points after coordinate transformation in the gradient intensity map using interpolation includes: Determine the pixels in the neighborhood surrounding each mapping point; Using any one of bilinear interpolation, cubic interpolation, or B-spline interpolation, the gradient intensity value at each mapping point is calculated based on the pixels in the neighborhood surrounding each mapping point.

6. The method according to any one of claims 1-4, characterized in that, After sorting according to the sum of the gradient intensity values ​​corresponding to each vertex, the method further includes: The optimal point, the second worst point, and the worst point are determined based on the sorting results. The optimal point has the largest sum of gradient strength values, the worst point has the smallest sum of gradient strength values, and the second worst point has a sum of gradient strength values ​​between the optimal point and the worst point. Calculate the centroids of the vertices of the simplex except for the worst-case point; Calculate the reflection point of the worst point relative to the centroid, and calculate the sum of the gradient intensity values ​​of the reflection point; If the sum of the gradient intensity values ​​of the reflection points is greater than the sum of the gradient intensity values ​​of the optimal points, then the expansion point is calculated, and the worst point is replaced by the expansion point or the point with the larger sum of gradient intensity values ​​among the reflection points. If the sum of the gradient intensity values ​​of the reflection points is between the sum of the gradient intensity values ​​of the second worst points and the sum of the gradient intensity values ​​of the best points, then the worst point is replaced by the reflection point. If the sum of the gradient intensity values ​​of the reflection points is greater than the sum of the gradient intensity values ​​of the worst point and less than or equal to the sum of the gradient intensity values ​​of the second worst point, then calculate the outer contraction point and replace the worst point with the outer contraction point; If the sum of the gradient intensity values ​​of the reflection points is less than or equal to the sum of the gradient intensity values ​​of the worst point, then the inner contraction point is calculated and the worst point is replaced by the inner contraction point.

7. A sub-pixel template matching device based on nonlinear optimization, characterized in that, include: The image acquisition unit is used to acquire the template image and the target image; The feature gradient generation unit is used to extract edge feature points from the template image and the target image respectively, to obtain a template feature point set and a target feature point set, and to construct a gradient intensity map, which is used to characterize the gradient intensity at each pixel position of the target image; An initial matching unit is used to perform coarse matching based on the template feature point set and the target feature point set to determine the initial position and initial angle of the template image in the target image; The pose transformation unit is used to perform coordinate transformation on the template feature point set according to the current matching pose composed of the sum of the initial position and the position offset, and the sum of the initial angle and the angle offset, wherein the initial values ​​of the position offset and the angle offset are zero. The gradient interpolation unit is used to obtain the gradient intensity values ​​at multiple mapping points after the coordinate transformation in the gradient intensity map by interpolation. An optimization output unit is used to iteratively update the position offset and the angle offset as optimization variables, with the goal of maximizing the sum of gradient intensity values, until a convergence condition is met, and output the sub-pixel level target position and target angle. The iterative update of the position offset and the angle offset includes: constructing an optimization space using the position offset and the angle offset as optimization variables, and constructing a simplex centered on an initial value. Each vertex of the simplex is used to calculate the corresponding sum of gradient intensity values ​​in each iteration. The simplex values ​​are sorted according to their sums. If the standard deviation of the sum of gradient intensity values ​​at each vertex of the simplex is less than a preset convergence threshold or the maximum number of iterations is reached, the iteration stops. The preset convergence threshold and the maximum number of iterations are the criteria for determining the convergence condition.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store one or more programs and configured to be executed by the processor, the programs including instructions for performing the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program for storing electronic data interchange is provided, wherein the computer program causes a computer to perform the method as described in any one of claims 1-6.

Images