A hydraulic pump parts tracking and registration method based on model-image joint perception

Abstract

The present invention discloses a hydraulic pump parts tracking and registration method based on joint perception of model and image. The method comprises collecting a photo stream of the hydraulic pump parts, obtaining a point cloud model from the photo stream, denoising the point cloud model and optimizing the point cloud density before storing it in a database as the original point cloud; using a calibrated camera to collect continuous frame images of the target being tracked in the hydraulic pump assembly environment in real time, and constructing a two-stage neighborhood hierarchical search mechanism and a three-factor feature weighted fusion based on a fast point feature histogram algorithm to extract feature points of the original point cloud in the database. In terms of time performance, the present invention can reduce processor computing costs and achieve real-time processing; and in terms of registration accuracy, through the effective fusion of point cloud features and visual features, combined with the nested loop mechanism of the pose estimation algorithm, the overall operating efficiency of the algorithm is improved, thereby enhancing the accuracy and robustness of the overall matching.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing, and in particular to a hydraulic pump part tracking and registration method based on model image joint perception. Background Art

[0002] Augmented reality research encompasses a wide range of areas. From a technical architecture perspective, it encompasses key areas such as camera calibration, tracking and registration, and scene fusion and display technologies. The core of an augmented reality system lies in integrating virtual information with the real scene. Tracking and registration technology calculates the camera's position relative to the real scene, providing spatial position and posture reference data for virtual-reality fusion.

[0003] Since some parts of hydraulic pumps have similar appearances, tracking and registration methods based on single modalities such as traditional two-dimensional images or identification have poor environmental adaptability, high hardware dependence, and high computational complexity, making it difficult to achieve efficient and stable tracking and registration. On the other hand, strategies that can achieve relatively stable tracking and registration, such as high-precision laser scanning, are too expensive. Therefore, designing a low-cost, efficient, and stable hydraulic pump part tracking and registration method that simultaneously guarantees real-time and robustness has become a technical problem that needs to be solved urgently. It is of great significance to improving the automation level and quality control of hydraulic pump assembly. Summary of the Invention

[0004] The purpose of the present invention is to provide a hydraulic pump parts tracking and registration method based on model image joint perception to solve the technical problems raised in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] Offline stage:

[0007] S1. Collect a photo stream of hydraulic pump parts and obtain a point cloud model from the photo stream. De-noise the point cloud model, optimize the point cloud density, and then store it in the database as the original point cloud.

[0008] Online stage:

[0009] S2. A calibrated camera is used to capture continuous frame images of the target in the hydraulic pump assembly environment in real time. A two-stage neighborhood hierarchical search mechanism and three-factor feature weighted fusion are constructed based on the fast point feature histogram algorithm to extract feature points from the original point cloud in the database.

[0010] S3, performing 3D-2D feature point matching on the feature points extracted from the database and the feature points of the continuous frame images of the tracked target obtained in real time to achieve dynamic coupling;

[0011] S4. Update the target pose based on the matching results, estimate the transformation relationship between the original point cloud in the database and the continuous frame images of the tracked target obtained in real time, and use the pose estimation algorithm to update the camera rotation and translation information external parameters to estimate the pose of the tracked target object;

[0012] S5. Corrective measures are introduced to optimize the estimated pose, and the optimized pose data is transmitted to the client using TCP for visual verification of the hydraulic pump assembly.

[0013] Furthermore, the specific steps of step S2 are:

[0014] S21, calculating the simplified point feature histogram SPFH of each point in the original point cloud within its first stage neighborhood;

[0015] S22, SPFH features corresponding to each point Perform L1 normalization to obtain the eigenvector corresponding to the point ;

[0016] S23, for each point in the original point cloud, further search for K nearest neighbor points in its second stage neighborhood, and fuse their SPFH features by weighted fusion , get the fast point feature histogram FPFH feature corresponding to each point;

[0017] S24, FPFH features corresponding to each point Perform L1 normalization to obtain the eigenvector corresponding to the point .

[0018] Furthermore, the calculation formula of step S21 is:

[0019] ,

[0020] In the above formula, , Represents the i-th point The corresponding first-stage neighborhood, where Represents the i-th point Relative to the jth point distance, is a distance threshold; is the angle mapping function, where K represents the number of histogram bins, 、 The i-th point With the jth point Connecting unit vectors Polar angle and azimuth on; is the linear distance weight function, , Indicates that the i-th point generates K-dimensional SPFH features.

[0021] Furthermore, the calculation formula of step S22 is:

[0022] ,

[0023] In the above formula, Represents the i-th point The FPFH characteristic value, Represents the i-th point The corresponding eigenvector, K represents the number of histogram bins.

[0024] Furthermore, the calculation formula of step S23 is:

[0025] ,

[0026] In the above formula, , Represents the mth point The corresponding second-stage neighborhood, Function expressed as a point As the center, find the distance The nearest K points; is the neighborhood feature weight coefficient of the first stage; is the Gaussian distance weight function, where Neighborhood points with dot distance, is the smoothing parameter; For each neighbor point After normalized SPFH features, Indicates that the i-th point generates K-dimensional FPFH features.

[0027] Furthermore, the calculation formula of step S24 is:

[0028] ,

[0029] In the above formula, Represents the i-th point The FPFH characteristic value, Represents the i-th point The corresponding eigenvector, K represents the number of histogram bins.

[0030] Furthermore, the specific steps of step S3 are:

[0031] S31, converting the coordinate unit of each 3D point in the extracted feature points into a homogeneous coordinate form;

[0032] S32, projecting the 3D point in homogeneous coordinate form onto a 2D plane, and normalizing the pixel coordinates projected onto the 2D plane;

[0033] S33, mapping the normalized projection point coordinates to the pixel coordinate system of the actual image, and performing coordinate transformation based on the focal length and image center coordinates in the camera intrinsic parameters;

[0034] S34. Calculate the Euclidean distance between the projection points mapped to the pixel coordinate system of the actual image and the feature points extracted from the continuous frame images of the tracking target, and determine whether the calculation result is less than a preset threshold. If so, the corresponding feature points are matched to achieve dynamic coupling of 3D-2D feature points.

[0035] Furthermore, in step S3, the feature points of the continuous frame images of the tracked target are acquired in real time using the ORB algorithm, the pose estimation algorithm in step S4 uses E-PnP, and the correction measure in step S5 uses the Levenberg-Marquardt algorithm.

[0036] Beneficial effects:

[0037] 1. The present invention has certain advantages in terms of real-time performance and robustness. In terms of time performance, it can reduce the computing cost of the processor and realize real-time processing. In terms of registration accuracy, through the effective fusion of point cloud features and visual features, a deep improvement based on the fast point feature histogram FPFH algorithm is proposed. A two-stage neighborhood search strategy and a three-factor feature weighted fusion method are adopted, and combined with the nested loop mechanism of the pose estimation algorithm, the overall operating efficiency of the algorithm is improved, thereby reducing the computing cost and improving the accuracy and robustness of the overall matching. Overall, the present invention has shown good results in multi-angle performance evaluations, verifying its practical application potential and technological advancement in complex scenarios.

[0038] 2. The present invention proposes a deep improvement based on the fast point feature histogram (FPFH) algorithm, constructs a two-stage neighborhood hierarchical search mechanism, and realizes a coarse-to-fine feature learning process through a hierarchical local area feature extraction and global feature fusion strategy; combined with feature weighted fusion technology, it effectively suppresses redundant information interference and enhances the expression ability of key features. While ensuring the accuracy of feature extraction, the computational complexity is reduced by about 20%-50%, thereby improving the robustness and expression ability of the extracted feature points to the local features of the three-dimensional point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions implemented in the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 This is a flow chart of the hydraulic pump parts tracking and registration method based on model-image joint perception of the present invention;

[0041] Figure 2 This is a framework diagram of the hydraulic pump parts tracking and registration method based on model-image joint perception of the present invention;

[0042] Figure 3 The present invention uses structure-from-motion technology to construct a point cloud model of hydraulic pump parts and extract characteristic points.

[0043] Figure 4 This is a diagram of the hydraulic pump parts position data transmission system of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] like Figure 1-Figure 4 As shown, the present invention provides a hydraulic pump parts tracking and registration method based on model image joint perception, and the specific steps are as follows:

[0046] In the offline stage, a photo stream of the hydraulic pump parts is collected, and a point cloud model is obtained from the photo stream. The point cloud model is processed and optimized and then stored in the database as the original point cloud. In this embodiment, the Structure from Motion (SFM) technology is used to generate a point cloud model of the hydraulic pump parts. At the same time, the data is transmitted to the data layer through the network layer to prepare for subsequent tracking and registration.

[0047] In this embodiment, in order to further illustrate step S1, the specific steps are as follows:

[0048] S11. Use a multi-camera array to synchronously capture a stream of photos of hydraulic pump parts, denoise the captured images, and use software such as Reality Capture to create an initial sparse point cloud model using SFM technology.

[0049] S12. Use professional 3D data processing software such as Meshlab to accurately identify and remove various noise points in the initial sparse point cloud model;

[0050] S13. Optimize the point cloud density using the ICP algorithm, and store the optimized point cloud model in the database.

[0051] In the online stage, a calibrated camera is used to capture continuous frame images of the target being tracked in the hydraulic pump assembly environment. A two-stage neighborhood hierarchical search mechanism and a three-factor feature weighted fusion are constructed based on the fast point feature histogram algorithm to extract feature points from the original point cloud in the database. These feature points can subsequently be used in many key research links, such as part recognition, matching, 3D reconstruction, and performance analysis. In terms of hardware requirements, only an ordinary RGB camera is required to complete the relevant tasks, reducing the cost-effectiveness of the implementation process.

[0052] To ensure the accuracy of 3D registration and precisely obtain the camera intrinsics and distortion parameters, the Zhang Zhengyou calibration method is first used. This method is based on a planar checkerboard. Normally, calibration requires only a planar calibration object (usually a checkerboard) to capture several images at different positions and postures.

[0053] In this embodiment, in order to further illustrate step S2, the specific steps are as follows:

[0054] Most current point cloud feature extraction technologies rely on a single-stage adaptive radius search framework, which presents computational efficiency bottlenecks. This embodiment proposes a deep improvement based on the Fast Point Feature Histogram (FPFH) algorithm, constructing a two-stage neighborhood hierarchical search mechanism. This mechanism utilizes a hierarchical local region feature extraction and global feature fusion strategy to achieve a coarse-to-fine feature learning process. Combined with weighted feature fusion technology, this effectively suppresses redundant information interference and enhances the expressiveness of key features. While maintaining feature extraction accuracy, this approach reduces computational effort by approximately 20%-50%, improving the robustness and expressiveness of the extracted feature points for the local features of the 3D point cloud.

[0055] S21. Calculate the simplified point feature histogram SPFH of each point in the original point cloud within its first-stage neighborhood. The calculation formula is:

[0056] ,

[0057] In the above formula, , Represents the i-th point The corresponding first-stage neighborhood, where Represents the i-th point Relative to the jth point distance, Is a distance threshold used to define which points belong to Neighborhood range; is the angle mapping function, where K represents the number of histogram bins, 、 The i-th point With the jth point Connecting unit vectors Polar angle and azimuth on; is the linear distance weight function, , and finally , each point in the original point cloud generates a 64-dimensional SPFH feature, that is, .

[0058] S22, SPFH features corresponding to each point Perform L1 normalization to obtain the eigenvector corresponding to the point , the calculation formula is:

[0059] ,

[0060] In the above formula, Represents the i-th point SPFH characteristics, Represents the i-th point The corresponding eigenvector, K represents the number of histogram bins.

[0061] S23, for each point in the original point cloud, further search for K nearest neighbor points in its second stage neighborhood, and fuse their SPFH features by weighted fusion , get the fast point feature histogram FPFH feature corresponding to each point, the calculation formula is:

[0062] ,

[0063] In the above formula, , Represents the mth point The corresponding second-stage neighborhood, Function expressed as a point As the center, find the distance The nearest K points; is the neighborhood feature weight coefficient of the first stage; is the Gaussian distance weight function, where Neighborhood points with dot distance, is the smoothing parameter; For each neighbor point After the normalized SPFH features, we finally get , each point generates a 64-dimensional FPFH feature, that is .

[0064] S24, FPFH features corresponding to each point Perform L1 normalization to obtain the eigenvector corresponding to the point , the calculation formula is:

[0065] ,

[0066] In the above formula, Represents the i-th point The FPFH characteristic value, Represents the i-th point The corresponding eigenvector, K represents the number of histogram bins.

[0067] S3, performing 3D-2D feature point matching on the feature points extracted from the database and the feature points of the continuous frame images of the tracked target obtained in real time using the ORB algorithm to achieve dynamic coupling;

[0068] In this embodiment, in order to further illustrate step S3, the specific steps are as follows:

[0069] S31. Convert the coordinate units of each 3D point in the extracted feature points to ensure the accuracy and consistency of the calculation; assuming that the original point cloud coordinate unit is centimeters, and the subsequent calculation needs to use meters as the unit, the unit conversion is achieved through a simple numerical transformation operation, as shown below. Suppose the original 3D point coordinates are , the converted coordinates are , the conversion formula is:

[0070] ,

[0071] In order to calculate the projection of a 3D point, the 3D point coordinates must first be converted to homogeneous coordinates; the pre-initialized projection matrix is ​​used to perform matrix multiplication with the 3D point in homogeneous coordinate form. Suppose the 3D point coordinates are , converted to homogeneous coordinates , the formula for constructing homogeneous coordinates is:

[0072] ,

[0073] Where P is a The coordinate matrix of is a The second coordinate matrix of .

[0074] S32. Project the 3D point in homogeneous coordinate form onto the 2D plane. Let the projection matrix be , the projected coordinates are , then the projection operation formula is:

[0075] ,

[0076] in is a The projection matrix, is a The coordinate matrix of .

[0077] Since the third dimension of the projected coordinates contains depth information, when converting them into pixel coordinates of the 2D image plane, the pixel coordinates projected onto the 2D plane need to be normalized. Let the projected coordinates be (n points, shape (3,n)), the normalized coordinates are (Shape is (2,n)), the normalization formula is:

[0078] ,

[0079] S33, mapping the normalized projection point coordinates to the pixel coordinate system of the actual image; performing coordinate transformation based on the focal length in the camera intrinsic parameters and the image center coordinates, so as to facilitate subsequent matching and visualization with the feature points in the image. Assume that the focal length in the camera intrinsic parameters is , , the image center coordinates are , , the image size is , the normalized coordinates Convert to pixel coordinates , the calculation formula is as follows:

[0080] ,

[0081] S34, calculate the Euclidean distance between the projection points mapped to the pixel coordinate system of the actual image and the feature points extracted from the continuous frame images of the tracking target, and determine whether the calculation result is less than a preset threshold. If so, the corresponding feature points are matched to achieve dynamic coupling of 3D-2D feature points; let the coordinates of a certain projection point projected onto the 2D image plane be , the coordinates of the ORB feature points extracted from the image are , the formula for calculating the Euclidean distance d between them is:

[0082] ,

[0083] when When , the two points are considered to match. Indicates the preset threshold.

[0084] The present invention traverses the projected 2D feature points and the ORB feature points in the image through nested loops, calculates the Euclidean distance between them, and when the distance is less than a set threshold, the two points are considered to be a pair of matching points, and their index combination is added to the list.

[0085] S4. Update the target pose based on the matching situation, estimate the transformation relationship between the original point cloud in the database and the continuous frame images of the tracked target obtained in real time, use the E-PnP pose estimation algorithm to update the camera's external parameters (rotation and translation information), and estimate the pose of the tracked target object to achieve the effect of real-time tracking of the target object and update its projected position in the image.

[0086] The present invention updates the target pose based on the results of the feature matching described above, estimates the external parameters of the object relative to the camera through the PnP algorithm, and then updates the projection matrix to reflect the real-time pose changes of the object in the camera's field of view. A feature matching distance threshold is set to determine whether the projected 3D point matches the ORB feature point extracted from the 2D image. E-PnP is to know the coordinates of multiple 3D spatial points in the world coordinate system and their corresponding projection coordinates on the 2D image plane (usually at least 5 pairs of matching points are required for effective pose estimation). When the number of matching point pairs is greater than or equal to 5, the corresponding 3D point coordinates and 2D image point coordinates are extracted from the matching results. After collecting a sufficient number of matching point pairs and knowing the camera's intrinsic parameters, the pose (rotation and translation) of the object relative to the camera is solved.

[0087] This embodiment establishes a series of equations based on the perspective projection model to solve the pose parameters:

[0088] Assume that a point in space The coordinates in the camera coordinate system are , the coordinate transformation between them is performed through the rotation matrix R and the translation vector t, as follows:

[0089] ,

[0090] where R is a The rotation matrix, t is a The translation vector of .

[0091] The camera intrinsic parameter matrix K obtained by the above calibration is used to convert the point in the camera coordinate system Projecting onto the 2D image plane to get pixel coordinates , the projection relationship is as follows:

[0092] ,

[0093] Where S is a scale factor.

[0094] For each 3D-2D point pair , expanding the above projection relationship can be obtained:

[0095] ,

[0096] Will Substituting into the equations we get a linear system , use singular value decomposition (SVD) to obtain the rotation matrix R and translation vector t.

[0097] S5. In the event of target loss, corrective measures are introduced to optimize the estimated pose. The optimized pose data is transmitted to the Unity client using TCP (Transmission Control Protocol) for visual verification of the hydraulic pump assembly. Virtual-reality fusion is performed on the client interface based on the pose data for visual verification of the hydraulic pump assembly to avoid cumulative errors as the tracking registration time increases.

[0098] The corrective measures are as follows: Use the Levenberg-Marquardt algorithm to perform nonlinear iterative optimization on the estimated pose, and finally obtain a more accurate rotation matrix R and translation vector t, and define the reprojection error The formula is as follows:

[0099] ,

[0100] In the above formula, Represents the process of projecting a 3D point onto a 2D image plane, They represent the horizontal and vertical coordinates of the pixel obtained by projecting the point in the camera coordinate system onto the 2D image plane when estimating the pose parameters.

[0101] The new rotation matrix R and translation vector t reflect the object's latest pose relative to the camera in the current frame. This matrix can then be used to project the 3D model's points onto the next frame's image plane for a new round of feature matching and pose update. This optimized pose data is then transmitted to the Unity client, where it is used in the Unity Game interface to perform virtual-reality fusion for visual verification of hydraulic pump assembly.

[0102] The present invention adopts a classic client-server architecture model to design a two-way communication system. Within this architecture, roles are clearly defined: the server, acting as a data producer, is responsible for generating and transmitting data; the client, acting as a receiver, is responsible for updating and processing pose data. The server implements an asynchronous I / O multiplexing mechanism based on the select module. This mechanism enhances the concurrent processing capabilities of the proposed method for tracking and registering hydraulic pump parts based on joint model-image perception. It manages multiple concurrent connections, ensuring rapid and orderly data transmission, enhanced by multimodal data, and thus enhancing the system's ability to cope with complex and changing environments.

[0103] At the protocol layer, this invention uses a fixed-length header (4 bytes) to identify the payload length, followed by the JSON-serialized application data. This design accurately identifies data frame boundaries, preventing confusion and misalignment during data transmission. Furthermore, the JSON-serialized data format supports flexible expansion of structured data, leaving ample room for future functionality expansion and data structure optimization.

[0104] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0105] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A hydraulic pump parts tracking and registration method based on model image joint perception, characterized in that: The following steps are involved: Offline stage: S1. Collect a photo stream of hydraulic pump parts and obtain a point cloud model from the photo stream. De-noise the point cloud model, optimize the point cloud density, and then store it in the database as the original point cloud. Online stage: S2. A calibrated camera is used to capture continuous frame images of the target in the hydraulic pump assembly environment in real time. A two-stage neighborhood hierarchical search mechanism and three-factor feature weighted fusion are constructed based on the fast point feature histogram algorithm to extract feature points from the original point cloud in the database. S3, performing 3D-2D feature point matching on the feature points extracted from the database and the feature points of the continuous frame images of the tracked target obtained in real time to achieve dynamic coupling; S4. Update the target pose based on the matching results, estimate the transformation relationship between the original point cloud in the database and the continuous frame images of the tracked target obtained in real time, and use the pose estimation algorithm to update the camera rotation and translation information external parameters to estimate the pose of the tracked target object; S5. Introduce corrective measures to optimize the estimated pose, and use TCP to transmit the optimized pose data to the client for visual verification of the hydraulic pump assembly; The specific steps of step S2 are: S21, calculating the simplified point feature histogram SPFH of each point in the original point cloud within its first stage neighborhood; S22, SPFH features corresponding to each point Perform L1 normalization to obtain the eigenvector corresponding to the point ; S23, for each point in the original point cloud, further search for K nearest neighbor points in its second stage neighborhood, and fuse their SPFH features by weighted fusion , get the fast point feature histogram FPFH feature corresponding to each point; S24, FPFH features corresponding to each point Perform L1 normalization to obtain the eigenvector corresponding to the point ; The calculation formula of step S21 is: , In the above formula, , Represents the i-th point The corresponding first-stage neighborhood, where Represents the i-th point Relative to the jth point distance, is a distance threshold; is the angle mapping function, where K represents the number of histogram bins, 、 The i-th point With the jth point Connecting unit vectors Polar angle and azimuth on; is the linear distance weight function, , Indicates that the i-th point generates K-dimensional SPFH features; The calculation formula of step S22 is: , In the above formula, Represents the i-th point The SPFH characteristics, Represents the i-th point The corresponding eigenvector, K represents the number of histogram bins; The calculation formula of step S23 is: , In the above formula, , Represents the mth point The corresponding second-stage neighborhood, Function expressed as a point As the center, find the distance The nearest K points; is the neighborhood feature weight coefficient of the first stage; is the Gaussian distance weight function, where Neighborhood points with dot distance, is the smoothing parameter; For each neighbor point After normalized SPFH features, Indicates that the i-th point generates K-dimensional FPFH features; The calculation formula of step S24 is: , In the above formula, Represents the i-th point The FPFH characteristic value, Represents the i-th point The corresponding eigenvector, K represents the number of histogram bins.

2. The hydraulic pump parts tracking and registration method according to claim 1, characterized in that: The specific steps of step S3 are: S31, converting the coordinate unit of each 3D point in the extracted feature points into a homogeneous coordinate form; S32, projecting the 3D point in homogeneous coordinate form onto a 2D plane, and normalizing the pixel coordinates projected onto the 2D plane; S33, mapping the normalized projection point coordinates to the pixel coordinate system of the actual image, and performing coordinate transformation based on the focal length and image center coordinates in the camera intrinsic parameters; S34. Calculate the Euclidean distance between the projection points mapped to the pixel coordinate system of the actual image and the feature points extracted from the continuous frame images of the tracking target, and determine whether the calculation result is less than a preset threshold. If so, the corresponding feature points are matched to achieve dynamic coupling of 3D-2D feature points.

3. The hydraulic pump parts tracking and registration method according to claim 1, characterized in that: In step S3, the feature points of the continuous frame images of the tracked target are acquired in real time using the ORB algorithm. In step S4, the pose estimation algorithm is E-PnP. In step S5, the correction measure is Levenberg-Marquardt algorithm.

Images