A multi-level mapping method for spatial relationship of non-rigid body

Abstract

A multi-level mapping method for spatial relationship of non-rigid body, comprising the following steps: S1 inputting a detection graph and a template graph; S2 performing feature point detection and extraction on the template graph and the detection graph to form a current feature point set; S3 fitting to obtain a current spatial transformation matrix set between the detection graph and the template graph based on the current feature point set; S4 screening an optimal spatial transformation matrix for a target instance in a reference image from the current spatial transformation matrix set, and taking the optimal spatial transformation matrix as a spatial mapping relationship of the target instance between the detection graph and the template graph. The present application establishes multiple spatial corresponding relationships between the detection graph and the template graph, constructs a spatial transformation relationship set in a region, and then screens an optimal spatial transformation relationship for each target instance, thereby fundamentally eliminating the registration error problem caused by the deficiency of a single registration method and reducing the false detection and missed detection rate of the subsequent image registration link.

Description

Technical Field

[0001] This invention relates to the field of spatial relationship mapping technology, and in particular to a multi-level spatial relationship mapping method applicable to non-rigid bodies. Background Technology

[0002] In industrial visual inspection scenarios, it is generally necessary to obtain the corresponding spatial transformation relationship between the template image and the inspection image. This is used to calculate information such as the positions of key points in the target template image within the inspection image, thereby enabling subsequent applications such as target localization and appearance inspection. Correctly obtaining the corresponding spatial transformation relationship is a crucial prerequisite for ensuring the accuracy and stability of the inspection results. In existing technologies, affine transformations and perspective transformations are commonly used as tools to solve geometric registration problems.

[0003] Specifically, affine transformation is suitable for handling linear deformations of planar objects in three-dimensional space, such as rotation, translation, and uniform scaling. For example, patent document CN114445472A proposes a multi-step image registration algorithm based on affine transformation and template matching. It describes a method for obtaining the transformation matrix of the affine transformation between the actual image and the schematic diagram based on the correspondence of matching points, thus completing the affine transformation. This method estimates the transformation matrix through feature matching point pairs, effectively compensating for rigid deformations caused by mechanical positioning or slight perspective deviations, and is suitable for positioning detection of planar products. Perspective transformation methods are applicable to non-planar products. For example, patent document CN111191708A describes how calculating the transmission transformation matrix can accurately describe the projection mapping relationship between the template image and the detection image. Compared to the affine matrix, its advantage lies in its ability to accurately correct nonlinear deformations caused by camera perspective, thereby achieving better image alignment when detecting curved surfaces, complex parts, and other scenarios.

[0004] The core assumptions of the above-mentioned transformation relationships are that the region to be registered is a rigid body or an indeformable plane. This leads to limitations in existing spatial relationship mapping methods when facing complex real-world scenarios. In actual scenarios, most mechanical structures are not rigidly fixed. When there are non-rigid moving parts on the surface of the object being detected, and there are moving relationships between them resulting in the existence of local moving regions, or when the detection area contains independent movable parts, such as braking devices, traction motors, and shock absorbers in train vehicles, a single global spatial relationship mapping method cannot describe the inconsistent deformation of different regions. This causes the existing spatial relationship mapping method based on a single global registration method to fail. If applied, it will lead to registration errors, resulting in a large number of false positives and false negatives. Summary of the Invention

[0005] This invention proposes a multi-level spatial relationship mapping method applicable to non-rigid bodies, which dynamically selects the optimal spatial transformation relationship for target instances in non-rigid sub-regions, effectively solving the problem that traditional spatial relationship mapping methods based on global single registration fail when dealing with complex non-rigid structures.

[0006] The technical solution adopted by this invention to solve the problem is: A multi-level mapping method for spatial relationships applicable to non-rigid bodies includes the following steps: S1 Input detection image and template image; S2 performs feature point detection and extraction on the template image and the detection image to form the current feature point set; S3. Based on the current set of feature points, fit and obtain the current set of spatial transformation matrices between the detection map and the template map; S3.1 Fit the feature points in the current feature point set to obtain the current spatial transformation matrix between the current detection image and the template image, and proceed to step S3.2; S3.2 Determine whether the preset loop termination condition has been met; If the preset loop termination condition is not met, the feature point pairs used to generate the spatial transformation matrix are removed from the current feature point set, and the remaining feature point set is returned to step S3.1 as the new current feature point set. If the preset loop termination condition is met, the loop terminates and proceeds to step S4. The above iteration process generates M current space transformation matrices H i (i=1......M) form the current spatial transformation matrix set H; S4 In the current set of spatial transformation matrices H, the optimal spatial transformation matrix is ​​selected for the target instance in the reference image, and it is used as the spatial mapping relationship between the detection image and the template image for the target instance. The optimal spatial transformation matrix is ​​based on the heatmap F of M feature points in the target search region. i The feature response intensity is obtained through screening; the feature point heatmap F i Based on the current spatial transformation matrix H i The corresponding fitted inlier points on the reference image are generated, and the target search region is determined based on the target instance.

[0007] Step S4 includes: S4.1 Generate a set of feature point heatmaps based on the current set of spatial transformation matrices. Based on the current spatial transformation matrix H i The fitted interior points used to generate a feature point heatmap F representing the feature point distribution of the reference image are used. iHeatmap of M feature points F i Form a set of feature point heatmaps F; S4.2 Optimal spatial transformation matrix for selecting target instances S4.2.1 For the target instance P in the reference image j (j=1......Q), determine the target instance P j The corresponding target search area S j ; S4.2.2 For the heatmap F of the M feature points i In the target search area S j The response intensities of M features within the target instance P are compared. Based on preset rules, the current spatial transformation matrix corresponding to the feature point heatmap that meets the requirement in terms of feature response intensity is selected as the target instance P. j The optimal spatial transformation matrix.

[0008] In step S4.2.2, the preset rule is: normalize the heatmaps of each feature point, and take the current spatial transformation matrix corresponding to the feature point heatmap with the maximum feature response intensity in the target search area that exceeds the preset feature response intensity as the optimal spatial transformation matrix of the target instance.

[0009] In step S3.1, the feature points in the current feature point set are fitted to obtain the current spatial transformation matrix between the current detection image and the template image. The method is as follows: S3.1.1 Fit the current feature point set, calculate the spatial transformation matrix between the detection image and the template image, and obtain the feature point pair set used to generate the spatial transformation matrix; S3.1.2 Filter the set of feature points obtained in step S3.1.1; S3.1.3 Refit the spatial transformation matrix with the set of feature points retained after filtering, and use it as the current spatial transformation matrix obtained in this step; The filtering of the obtained feature point pair set in step S3.1.2 includes: calculating the intersection points between the lines connecting the feature point pairs in the feature point pair set, and removing the feature point pairs corresponding to the lines with a number of intersection points greater than or equal to a threshold Tc; and inputting the retained feature point pairs into step S3.1.3, where the threshold Tc is greater than or equal to 1.

[0010] The preset loop termination condition in step S3.2 is: the number of remaining feature point pairs is less than the preset number Ta, or the number of loops reaches the preset number of loops Tb.

[0011] In step S4, the target instance is determined to have its corresponding target search area in the following manner: For the target instance in the reference image, the target instance bounding box is magnified N times, and the magnified target instance bounding box is used as the target search area; The N≥1.

[0012] The feature point heatmap F in step S4 i The current spatial transformation matrix H i The summation of feature descriptor maps obtained from all fitted inlier points on the corresponding reference image; The characteristic response intensity of any point (x, y) in the feature point heatmap Fi is the current spatial transformation matrix H. i The sum of the feature response intensities of all fitted interior points on the corresponding reference image at (x,y).

[0013] The feature point heatmap is generated using the following method: Create a blank feature point heatmap with the same size as the reference image; For the current spatial transformation matrix H i For each fitted inlien point on the corresponding reference image, a Gaussian distribution is calculated centered on the corresponding fitted inlien point and accumulated onto the blank feature point heatmap. Finally, a heatmap with each fitted inlien point as the intensity center is formed on the blank feature point heatmap, thus creating the feature point heatmap F of the reference image. i .

[0014] The spatial transformation matrix between the detection image and the template image is a homography matrix.

[0015] The reference image is a template image.

[0016] The beneficial effects of this invention are as follows: Addressing the challenge of registering non-rigid structures, it establishes multiple spatial correspondences between the detection image and the template image. It introduces a feature heatmap to dynamically select the optimal spatial transformation relationship for target instances in non-rigid sub-regions, effectively solving the problem of traditional spatial relationship mapping methods based on global single registration failing when dealing with complex non-rigid structures. For complex detection objects with multiple components, especially those containing moving parts, considering that non-rigid bodies are composed of multiple rigid or near-rigid sub-regions, this invention constructs a set of spatial transformation relationships by region, and then selects the optimal spatial transformation relationship for each target instance. This fundamentally eliminates registration errors caused by the inadequacy of a single registration method, reducing the false detection and false negative rates in subsequent image registration stages.

[0017] This invention integrates multi-level regional feature points and thermal field maps to select the optimal spatial transformation relationship for the target to be matched, thereby significantly improving the registration accuracy of non-rigid bodies and solving the limitation that traditional geometric registration is only applicable to rigid bodies. At the same time, it uses a template image as the reference image to select the optimal spatial transformation matrix for the target instance in the template image, avoiding the problem that deep learning registration models require a large amount of pairing training data.

[0018] This invention expands the scope of visual inspection, enabling stable and accurate handling of non-rigid objects and complex assemblies with moving parts that are difficult to process using traditional spatial transformation methods. It is applied to the inspection of complex rail transit assemblies, such as precision instruments containing springs, rails, or linkage modules, braking devices in train vehicles, traction motors, and shock absorbers, providing reliable quality inspection under non-rigid conditions.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] A multi-level mapping system for spatial relationships applicable to non-rigid bodies, comprising: A 2D camera is used to acquire detection images and template images; The data processing module is used to detect and extract feature points in the detection image and template image, and to perform subsequent processing on the extracted feature points, including fitting the current set of spatial transformation matrices and filtering the optimal spatial transformation matrix for the target instance. A pre-trained deep learning model is used to process the detection map and template map to obtain the target instance.

[0024] Deep learning models are commonly used for instance segmentation, object detection, and object tracking.

[0025] A multi-level mapping method for spatial relationships applicable to non-rigid bodies, such as Figure 1 As shown, it includes the following steps: S1 Input detection image and template image; S2 performs feature point detection and extraction on the template image and the detection image to form the current feature point set; the current feature point set here includes the feature point set of the template image and the feature point set of the detection image. S3 Based on the current set of feature points, fits and obtains the current set of spatial transformation matrices between the detection map and the template map; S3.1 Fit the feature points in the current feature point set to obtain the current spatial transformation matrix between the current detection image and the template image, and proceed to step S3.2; S3.2 Determine whether the preset loop termination condition has been met; If the preset loop termination condition is not met, remove the feature point pairs used to generate the spatial transformation matrix from the current feature point set, and return to step S3.1 with the remaining feature point set as the new current feature point set; If the preset loop termination condition is met, the loop terminates and proceeds to step S4. The above iteration process generates M current space transformation matrices H i (i=1......M) form the current spatial transformation matrix set H; S4 In the current set of spatial transformation matrices H, the optimal spatial transformation matrix is ​​selected for the target instance in the reference image, and it is used as the spatial mapping relationship between the target instance in the detection map and the template map; the reference image is either the detection map or the template map, and in one embodiment, the template map is selected. The optimal spatial transformation matrix is ​​based on the heatmap F of M feature points in the target search region. i Feature response intensity was obtained through screening; feature point heatmap F i Based on the current spatial transformation matrix H i The fitted inlier points on the corresponding benchmark image are generated, and the target search region is determined based on the target instance.

[0026] The feature point heatmap represents the intensity of the influence of the feature point on its surrounding area, which generally decreases from the feature point as the center to the outside.

[0027] The influence intensity of the current feature point on its surroundings is characterized by a Gaussian function centered on that feature point. The Gaussian function is applied to all feature points corresponding to each spatial transformation matrix, and the summation is used to obtain the feature point heatmap corresponding to the current spatial transformation matrix.

[0028] The optimal spatial transformation matrix selected for each target instance in S4 is applied in subsequent registration. Specifically, the current set of spatial transformation matrices H contains {H1, H2, ..., H...} M There are a total of M current spatial transformation matrices. Through the filtering process in step S4, the target instance P in the reference image is determined. j(j=1......Q), obtain its corresponding optimal spatial transformation matrix H i Suppose there are Q1 target instances with optimal spatial transformation matrix H1, Q2 target instances with optimal spatial transformation matrix H2, ..., Q... M The optimal spatial transformation matrix for each target instance is H. M In the subsequent registration process, H1 is selected as the spatial mapping relationship between the detection map and the template map for these Q1 target instances, and so on.

[0029] There are two types of preset loop termination conditions: one is that the number of remaining feature point pairs is less than the preset number Ta; the other is that the number of loops reaches the preset number of loops Tb.

[0030] Fitted interior points refer to points that conform to the description of the current spatial transformation matrix.

[0031] A template image is a standard image used as a reference. A perfect, properly positioned picture of the part to be inspected is taken as a template image. An inspection image is a picture taken by the camera during the actual inspection process. The image quality is affected by factors such as the shooting angle and the on-site environment.

[0032] In one specific embodiment, step S4 includes: S4.1 Generate a set of feature point heatmaps based on the current set of spatial transformation matrices. Based on the current spatial transformation matrix H i The fitted interior points used to generate a feature point heatmap F representing the feature point distribution of the benchmark image are used. i Heatmap of M feature points F i Form a set of feature point heatmaps F; S4.2 Optimal spatial transformation matrix for selecting target instances S4.2.1 For target instance P in the reference image j (j=1......Q), determine the target instance P j The corresponding target search area S j ; S4.2.2 Heatmap F for M feature points i In the target search area S j The intensity of M feature responses within the target instance is compared. Based on preset rules, the current spatial transformation matrix corresponding to the feature point heatmap that meets the requirements in terms of feature response intensity is selected as the target instance P. j The optimal spatial transformation matrix.

[0033] For each target instance labeled or identified in the baseline image, the above operation is performed to select the optimal spatial transformation matrix.

[0034] Feature point heatmap Fi It reflects the feature response regions and intensities between the baseline image (detection image or template image) and another image (template image or detection image).

[0035] In a specific embodiment, in step S4.2.2, the preset rule is: normalize the heatmap of each feature point, and take the current spatial transformation matrix corresponding to the feature point heatmap with the maximum feature response intensity in the target search area that exceeds the preset feature response intensity as the optimal spatial transformation matrix of the target instance.

[0036] Normalization refers to unifying the frequency of different feature point heatmaps. Frequency unification involves taking the minimum and maximum values ​​of the number of interior points and then normalizing them to 0-255. The purpose is to improve the visualization of the feature point heatmap. Global normalization is performed on the feature points generated by fitting interior points to the multi-round spatial transformation matrix to ensure uniform scale across different rounds.

[0037] The target instance in step S4.2.1 can be obtained through instance segmentation, target detection, and target tracking. For the template graph, the target instance can also be obtained through annotation. After obtaining the target instance, the instance bounding box or an expanded instance bounding box is used as the target search area. In a specific embodiment, the target instance is obtained through an instance segmentation model, specifically using the YOLO11 network model.

[0038] The preset feature response intensity is set according to the actual situation. When the maximum feature response intensity does not exceed the preset feature response intensity, the target search area is expanded according to the preset parameters until the number of iterations is greater than the threshold number or exceeds the preset feature response intensity, at which point the process exits.

[0039] In a specific embodiment, in step S3.1, the feature points in the current feature point set are fitted to obtain the current spatial transformation matrix between the current detection image and the template image. The method is as follows: S3.1.1 Fit the current set of feature points, calculate the spatial transformation matrix between the detection image and the template image, and obtain the set of feature point pairs used to generate the spatial transformation matrix; S3.1.2 Filter the set of feature points obtained in step S3.1.1; S3.1.3 Refit the spatial transformation matrix with the set of feature points retained after filtering, and use it as the current spatial transformation matrix obtained in this step; In step S3.1.2, the obtained feature point pair set is filtered, which includes: calculating the intersection points between the lines connecting the feature point pairs in the feature point pair set, and removing the feature point pairs corresponding to the lines whose number of intersection points is greater than or equal to the threshold Tc; the retained feature point pairs are then input into step S3.1.3, where the threshold Tc is greater than or equal to 1.

[0040] In step S3.1.1, the findHomography function is used and the RANSAC method is selected to fit the transmission matrix. The 2D pixel coordinates of two sets of corresponding points are input, and the general findHomography function is called. This function can find and return the perspective transformation matrix between the source plane and the target plane based on the correspondence between the points.

[0041] Specifically, feature points of two images of the same real object taken at different times are extracted using the SURF algorithm. After inlier feature pairs are obtained by FLANN matching and RANSAC algorithm, the homography matrix is ​​estimated using the cv2.findHomography function of the OpenCV library. This matrix is ​​a 3×3 homogeneous projection transformation matrix, which represents the projection transformation relationship from the pixel coordinates of the first image to the corresponding pixel coordinates of the second image.

[0042] In step S3.1.3, the spatial transformation matrix is ​​refitted using the least squares method. Since step S3.1.3 refits the filtered interior points, RANSAC filtering is not required.

[0043] The feature point pair connection is the connection between a set of mutually matching feature points in the detection map and the template map.

[0044] Filtering also considers factors such as the angle and distance between feature point pairs, and the specific methods are as follows: Compare the angle value of each set of feature point matching pairs with the overall average angle, and delete feature point pairs whose absolute difference exceeds the angle threshold. Compare the distance value of each set of feature point matching pairs with the overall average distance, and delete feature point pairs whose absolute difference exceeds the distance threshold; The threshold Tc is set according to the actual situation. During implementation, considering that a line connecting an abnormal point may intersect with multiple lines, and two relatively parallel lines may intersect, the threshold Tc is set to an integer greater than 1.

[0045] In a specific embodiment, the preset loop termination condition in step S3.2 is: the number of remaining feature point pairs is less than a preset number Ta, or the number of loops reaches a preset number of loops Tb.

[0046] The preset quantity Ta and preset number of cycles Tb are set according to the specific application scenario.

[0047] In a specific embodiment, in step S4, the target instance is determined to have its corresponding target search area in the following manner: For a target instance in the baseline image, the target instance bounding box is magnified N times, and the magnified target instance bounding box is used as the target search area. N≥1.

[0048] When N=1, it means that the target instance box is not enlarged.

[0049] In one embodiment, the initial N0 is set to 1.5, which means that after the target instance box is enlarged by 1.5 times, the feature response intensity is counted in the enlarged target instance box. If the maximum feature response intensity value does not exceed the preset feature response intensity, the target instance box is enlarged again according to the preset parameters, and the target instance box is enlarged to N1 times, N2 times, etc., in turn, or the search box is enlarged to n1 pixels, n2 pixels, etc., in turn, as a new target search area, until the number of enlargements is greater than the threshold number or the counted feature response intensity exceeds the preset feature response intensity, at which point the process exits.

[0050] In a specific embodiment, the feature point heatmap F in step S4 i The current spatial transformation matrix H i The summation of feature descriptor maps obtained from all fitted inliers on the corresponding baseline image; Feature point heatmap F i The characteristic response intensity at any point (x, y) in the space is the current spatial transformation matrix H. i The sum of the feature response intensities of all fitted interior points on the corresponding baseline image at (x,y).

[0051] The feature description map is generated by the feature response intensity function centered at each interior point.

[0052] Assuming the standard characteristic response intensity function centered at a single interior point is G(x,y), for some interior point (x... i y i The characteristic response intensity function at (x,y) is G(xx). i yy i The final characteristic response intensity at (x,y) is ∑G.

[0053] The feature response intensity within the target search area is the sum of the final feature response intensities of all points (x,y) within the target search area.

[0054] The following method is used to generate feature point heatmaps: Create a blank feature point heatmap with the same size as the baseline image; Based on the fitted interior points of the current spatial transformation matrix, a function representing the influence range of feature points is generated in the blank feature point heatmap; Discrete feature points are converted into continuous feature response maps, which serve as feature point heatmaps for the baseline image.

[0055] In one specific embodiment, the feature point heatmap is generated using the following method: Create a blank feature point heatmap with the same size as the baseline image; For the current spatial transformation matrix H i For each fitted inlien point on the corresponding reference image, a Gaussian distribution is calculated centered on the corresponding inlien point and accumulated in the blank feature point heatmap. Finally, a heatmap with each fitted inlien point as the intensity center is formed on the blank feature point heatmap, resulting in the feature point heatmap F of the reference image. i .

[0056] For each round of generated spatial transformation matrix, the pixel coordinates of the fitted interior points of the reference image are extracted. A Gaussian kernel density estimation method is used (kernel parameters are set to 500 and 100) to generate a single-channel Gaussian density heatmap (feature distribution heatmap) of the fitted interior points of the spatial transformation matrix. The pixel values ​​of this heatmap represent the spatial density of the fitted interior points of the corrected image in the corresponding region. The higher the density, the larger the pixel value, and vice versa.

[0057] Global normalization is performed on the Gaussian density heatmap of feature points generated by fitting interior points to the multi-round spatial transformation matrix, ensuring uniform brightness scale across heatmaps from different rounds. For each spatial transformation matrix H... i Regarding the H i The feature point heatmap F is obtained by summing the feature description maps of all fitted interior points. i Force max(F) i )=255,min(F i F = 0, linear distribution between max and min, F i =ΣG.

[0058] In one specific embodiment, the spatial transformation matrix between the detection image and the template image is a homography matrix.

[0059] In one specific embodiment, a template image reference is selected.

[0060] In a specific embodiment, in step S2, feature points are detected using the following method: the Surf algorithm is used to detect feature points. When the number of feature points detected by the Surf algorithm is insufficient, the SIFT algorithm is used to supplement the detection. The feature point results detected by the two algorithms are superimposed to form the current feature point set.

[0061] This implementation method is suitable for scenarios with many feature points, such as rail transit scenarios, which include various structures such as tracks, overhead contact lines, and carriages. These scenarios are characterized by rich feature point distribution and complex scene structures. The SURF algorithm can detect a sufficient number of feature points with high stability for such scenarios, and can meet the efficiency requirements of real-time detection in rail transit scenarios. For low-texture areas with insufficient feature points, such as smooth carriage surfaces or monochromatic track areas, the SIFT algorithm can supplement the detection of areas with missing feature points.

[0062] For this type of scenario, SURF can detect a large number of feature points and is stable. However, in some cases, SURF may not detect enough feature points. In this case, SIFT algorithm is used to supplement the detection. The feature vectors of the two algorithms are unified, which makes it easy to fuse and superimpose the detection results.

[0063] Compared with existing technologies, this invention addresses the challenge of registering non-rigid structures by establishing multiple spatial correspondences between the detection image and the template image. It introduces a feature heatmap to dynamically select the optimal spatial transformation relationship for target instances in non-rigid sub-regions, effectively solving the problem of traditional spatial relationship mapping methods based on global single registration failing when dealing with complex non-rigid structures. For complex detection objects with multiple components, especially those containing moving parts, considering that the non-rigid body is composed of multiple rigid or near-rigid sub-regions, this invention constructs a set of spatial transformation relationships by region, and then selects the optimal spatial transformation relationship for each target instance. This fundamentally eliminates registration errors caused by the inadequacy of a single registration method, reducing the false detection and false negative rates in subsequent image registration stages. By fusing multi-level regional feature points and heatmaps, this invention can select the optimal spatial transformation relationship for the target to be matched, significantly improving the registration accuracy of non-rigid bodies and overcoming the limitation that traditional geometric registration is only applicable to rigid bodies. Simultaneously, using the template image as the reference image, it selects the optimal spatial transformation matrix for target instances in the template image, avoiding the problem of deep learning registration models requiring a large amount of paired training data.

[0064] This invention expands the scope of visual inspection, enabling stable and accurate handling of non-rigid objects and complex assemblies with moving parts that are difficult to process using traditional spatial transformation methods. It is applied to the inspection of complex rail transit assemblies, such as precision instruments containing springs, rails, or linkage modules, braking devices in train vehicles, traction motors, and shock absorbers, providing reliable quality inspection under non-rigid conditions.

[0065] Furthermore, it should be understood that the above embodiments only illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A multi-level mapping method for spatial relationships applicable to non-rigid bodies, characterized in that, Includes the following steps: S1 Input detection image and template image; S2 performs feature point detection and extraction on the template image and the detection image to form the current feature point set; S3. Based on the current set of feature points, fit and obtain the current set of spatial transformation matrices between the detection map and the template map; S3.1 Fit the feature points in the current feature point set to obtain the current spatial transformation matrix between the current detection image and the template image, and proceed to step S3.2; S3.2 Determine whether the preset loop termination condition has been met; The preset loop termination condition is: the number of remaining feature point pairs is less than the preset number Ta, or the number of loops reaches the preset number of loops Tb. If the preset loop termination condition is not met, the feature point pairs used to generate the spatial transformation matrix are removed from the current feature point set, and the remaining feature point set is returned to step S3.1 as the new current feature point set. If the preset loop termination condition is met, the loop terminates and proceeds to step S4. The above iteration process generates M current space transformation matrices H i The current spatial transformation matrix set H is formed, where i=1......M; S4 In the current set of spatial transformation matrices H, the optimal spatial transformation matrix is ​​selected for the target instance in the reference image, and it is used as the spatial mapping relationship between the detection image and the template image for the target instance. The optimal spatial transformation matrix is ​​based on the heatmap F of M feature points in the target search region. i The characteristic response intensity is obtained by filtering according to preset rules; the feature point heatmap F i Based on the current spatial transformation matrix H i The corresponding fitted inlier points on the reference image are generated, and the target search region is determined based on the target instance; The reference image is a detection image or a template image; The feature point heatmap F i The current spatial transformation matrix H i The summation of feature description maps obtained from all fitted inliers on the corresponding reference image; the feature description map is generated by a feature response intensity function centered on the fitted inlier. The preset rule is as follows: normalize the heatmaps of each feature point, and take the current spatial transformation matrix corresponding to the feature point heatmap with the largest feature response intensity in the target search area that exceeds the preset feature response intensity as the optimal spatial transformation matrix of the target instance.

2. The method for multi-level mapping of spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, Step S4 includes: S4.1 Generate a set of feature point heatmaps based on the current set of spatial transformation matrices. Based on the current spatial transformation matrix H i The fitted interior points used to generate a feature point heatmap F representing the feature point distribution of the reference image are used. i Heatmap of M feature points F i Form a set of feature point heatmaps F; S4.2 Optimal spatial transformation matrix for selecting target instances S4.2.1 For the target instance P in the reference image j Determine the target instance P j The corresponding target search area S j Where j=1......Q; S4.2.2 For the heatmap F of the M feature points i In the target search area S j The response intensities of M features within the target instance P are compared. Based on the preset rules, the current spatial transformation matrix corresponding to the feature point heatmap that meets the requirement in terms of feature response intensity is selected as the target instance P. j The optimal spatial transformation matrix.

3. The method for multi-level mapping of spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, In step S3.1, the feature points in the current feature point set are fitted to obtain the current spatial transformation matrix between the current detection image and the template image. The method is as follows: S3.1.1 Fit the current feature point set, calculate the spatial transformation matrix between the detection image and the template image, and obtain the feature point pair set used to generate the spatial transformation matrix; S3.1.2 Filter the set of feature points obtained in step S3.1.1; S3.1.3 Refit the spatial transformation matrix with the set of feature points retained after filtering, and use it as the current spatial transformation matrix obtained in this step; The filtering of the obtained feature point pair set in step S3.1.2 includes: calculating the intersection points between the lines connecting the feature point pairs in the feature point pair set, and removing the feature point pairs corresponding to the lines with a number of intersection points greater than or equal to a threshold Tc; and inputting the retained feature point pairs into step S3.1.3, where the threshold Tc is greater than or equal to 1.

4. The multi-level mapping method for spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, In step S4, the target instance is determined to have its corresponding target search area in the following manner: For the target instance in the reference image, the target instance bounding box is magnified N times, and the magnified target instance bounding box is used as the target search area; The N≥1.

5. The method for multi-level mapping of spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, The characteristic response intensity of any point (x, y) in the feature point heatmap Fi is the current spatial transformation matrix H. i The sum of the feature response intensities of all fitted interior points on the corresponding reference image at (x,y).

6. The method for multi-level mapping of spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, The feature point heatmap is generated using the following method: Create a blank feature point heatmap with the same size as the baseline image; Based on the fitted interior points of the current spatial transformation matrix, a feature point influence range representation function is generated in the blank feature point heatmap, which serves as the feature point heatmap of the reference image.

7. A multi-level mapping method for spatial relationships applicable to non-rigid bodies as described in claim 6, characterized in that, The process of fitting interior points based on the current spatial transformation matrix and generating a feature point influence range representation function in the blank feature point heatmap includes: adjusting the current spatial transformation matrix H... i For each fitted inlien point on the corresponding reference image, a Gaussian distribution is calculated centered on the corresponding fitted inlien point and accumulated onto the blank feature point heatmap. Finally, a heatmap with each fitted inlien point as the intensity center is formed on the blank feature point heatmap, thus creating the feature point heatmap F of the reference image. i .

8. The method for multi-level mapping of spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, The spatial transformation matrix between the detection image and the template image is a homography matrix.

9. A multi-level mapping method for spatial relationships applicable to non-rigid bodies as described in claim 1, characterized in that, The reference image is a template image.

10. A multi-level mapping method for spatial relationships applicable to non-rigid bodies as described in claim 3, characterized in that, The filtering of the obtained feature point set in step S3.1.2 further includes: Compare the angle value of each pair of feature points with the overall average angle, and delete the feature point pairs whose absolute difference exceeds the angle threshold. The distance value of each pair of feature points is compared with the overall average distance, and feature point pairs with an absolute difference exceeding the distance threshold are deleted.

Images