Real-time positioning method and system for mechanical arm operation

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a real-time positioning method and system for mechanical arm operation, and the method comprises the steps: calculating an initial matching cost volume of a three-dimensional image pair; generating an adaptive aggregation path field according to the local structure tensor of the image, and determining an adaptive penalty term based on the structural strength; performing cost aggregation along the adaptive path field by using the adaptive penalty term to obtain an aggregation cost volume; and a disparity map is determined according to the aggregation cost volume, and the three-dimensional pose of the to-be-drilled target point is solved. According to the method, the aggregation path fitting the geometric structure of the scene is generated, and the smooth constraint is dynamically adjusted, so that the precision and robustness of three-dimensional positioning in a complex environment are remarkably improved, and reliable real-time pose guidance is provided for high-precision operation of the mechanical arm.

Description

Technical Field

[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to a real-time positioning method and system for robotic arm operations. Background Technology

[0002] In modern tunnel engineering, stable and reliable wireless communication is a core element for ensuring construction safety and operational efficiency. Leaky cables, as an effective signal coverage solution, are widely used in highway, railway, and subway tunnels. Traditional leaky cable installation mainly relies on manual labor for positioning, drilling, and support fixing. This method is not only labor-intensive and inefficient, but also poses high safety risks in complex tunnel environments. Therefore, using intelligent construction robotic arms for automated drilling operations has become an important development direction for improving the intelligence level of tunnel construction.

[0003] To automate drilling operations, intelligent construction robotic arms must be equipped with high-precision 3D vision perception systems to acquire real-time geometric information of the tunnel wall surface. The core task of this system is to accurately calculate the spatial 3D coordinates of the target point to be drilled and its surface normal vector, thereby guiding the drill bit at the end of the robotic arm to drill perpendicularly to the tunnel wall surface, ensuring the stability and proper installation of the cable support. Currently, non-contact 3D measurement technology based on binocular stereo vision has become the preferred solution for such guidance systems due to its ability to quickly acquire dense depth information of the scene.

[0004] However, tunnel walls are typically non-cooperative complex curved surfaces formed by shotcrete technology. These surfaces are uneven and have varying curvatures. Traditional stereo matching algorithms (such as semi-global block matching SGBM) mostly use fixed linear paths for cost aggregation. These paths cannot effectively fit the actual geometric contours of the tunnel wall. When the aggregated path traverses curved surfaces or areas of depth discontinuity, severe parallax mismatches occur, leading to a significant decrease in the accuracy of the 3D reconstruction model near key borehole locations. Simultaneously, the uneven distribution of texture on the tunnel wall surface means that traditional stereo matching algorithms use fixed smoothing penalties. In textured areas, this can lead to insufficient smoothing and ineffective noise suppression, while in areas with uniform texture, it can result in over-smoothing and loss of necessary structural details. This affects the accuracy of calculating the surface normal vectors at borehole points, making it difficult to meet the requirements of high-precision drilling by robotic arms. Summary of the Invention

[0005] To address the technical problem that existing stereo matching algorithms suffer from low positioning accuracy on complex curved surfaces due to fixed aggregation paths and penalty terms, thus failing to meet the high-precision operation requirements of robotic arms, this invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a real-time positioning method for robotic arm operations, comprising: A stereo image pair to be processed is acquired, and an initial matching cost volume is calculated. The initial matching cost volume consists of the matching costs between pixels in the stereo image pair under different disparities. The local structure tensor of each pixel is calculated based on one image in the stereo image pair, and an adaptive aggregation path field covering the image is generated based on the local structure tensor. The structural strength of each pixel is determined based on the local structure tensor, and an adaptive penalty term for cost aggregation is determined based on the structural strength. Cost aggregation is performed on the initial matching cost volume along each aggregation path in the adaptive aggregation path field using the adaptive penalty term to obtain an aggregated cost volume. A disparity map is determined based on the aggregated cost volume, and the three-dimensional pose of the target point to be drilled is determined based on the disparity map to guide the robotic arm drill bit to perform vertical drilling operations.

[0007] This invention analyzes the local structure tensor of an image to accurately identify the direction of the surface contour of a target object in the image. The adaptive aggregation path generated based on this information can closely fit the real geometric structure of the scene, ensuring that the cost aggregation process strictly adheres to the depth continuity assumption. This fundamentally solves the matching error problem caused by traditional fixed paths crossing depth boundaries and significantly improves the positioning accuracy on irregular curved surfaces.

[0008] Preferably, the matching cost is obtained by performing distortion correction and stereo correction on the stereo image pair; for each pixel in the corrected image, within a preset disparity range, the matching cost between the pixel and the corresponding pixel in the corrected image is calculated, and the matching cost is composed of a weighted average of the Census transform cost and the SAD cost.

[0009] This invention uses a weighted fusion of the Census transform cost, which is robust to changes in illumination, and the SAD cost, which is sensitive to grayscale details. This enables the calculation of the initial matching cost to effectively resist the interference of complex lighting in industrial environments, while preserving the fine texture information of the working surface, thus providing reliable raw data for subsequent cost aggregation.

[0010] Preferably, calculating the local structure tensor of each pixel based on one of the three-dimensional image pairs includes: calculating the horizontal and vertical gradients of each pixel in the image; and for each pixel, constructing its local structure tensor based on the horizontal and vertical gradients of each pixel in its neighborhood window.

[0011] Preferably, generating an adaptive aggregated path field covering the image based on the local structure tensor includes: performing eigenvalue decomposition on the local structure tensor of each pixel to obtain two eigenvalues ​​and corresponding eigenvectors; taking the eigenvector corresponding to the smallest eigenvalue among the two eigenvalues ​​as the local optimal structure direction of that pixel; starting from the boundary pixels of the image, extending into the image according to the local optimal structure direction of each pixel to generate a set of aggregated paths, constituting the adaptive aggregated path field.

[0012] This invention decomposes the local structure tensor into eigenvalues ​​and identifies the direction of the gentlest gradient change as the local optimal structure direction. The aggregated path field generated by this direction can extend along the structural edges such as the outline of objects in the image, ensuring that the cost aggregation is carried out in the neighborhood of pixels with similar depth values. This avoids erroneous smoothing across different depth planes and ensures the clarity and accuracy of the disparity map at the object edges.

[0013] Preferably, determining the structural strength of each pixel based on the local structural tensor of each pixel includes: performing eigenvalue decomposition on the local structural tensor of each pixel, and taking the largest eigenvalue among the two eigenvalues ​​obtained from the decomposition as the structural strength of that pixel.

[0014] Preferably, determining the adaptive penalty term for cost aggregation based on the structural strength includes: determining the structural strength weight of each pixel according to the structural strength of each pixel. In the formula, For pixels Structural strength weight; For pixels Structural strength; A coefficient used to control the sensitivity of the weights; It is a natural exponential function; an adaptive penalty term is constructed based on the structural strength weight of each pixel.

[0015] This invention quantizes the maximum eigenvalue of the local structure tensor into structural strength and generates structural strength weights accordingly, thereby achieving accurate evaluation of the texture complexity of local image regions. The structural strength weights are used to dynamically adjust the smoothing penalty term in the cost aggregation process, so that a larger smoothing constraint is applied to the strong structure region with clear contours to suppress noise, while a smaller constraint is applied to the flat region with simple texture to preserve details. This adaptive mechanism can effectively solve the drawbacks of fixed penalty terms and achieve an intelligent balance between noise suppression and detail preservation.

[0016] Preferably, along each aggregation path in the adaptive aggregation path field, the initial matching cost volume is aggregated using the adaptive penalty term to obtain an aggregated cost volume, including: determining the path aggregation cost of the aggregation path at different disparities at each pixel point; accumulating the path aggregation costs of all aggregation paths passing through each pixel point at the same disparity at that pixel point as the final aggregation cost of that pixel point at the corresponding disparity; and constructing an aggregated cost volume from the final aggregation costs of all pixels at all disparities.

[0017] Preferably, the path aggregation cost satisfies the expression: ; In the formula, For the target path at the pixel point Location, parallax Path aggregation cost at that time; For pixels At parallax The matching cost between the current pixel and the corresponding pixel in the right image; For the pixels on the target path The previous pixel; For the previous pixel At the same parallax The aggregation cost below; For the previous pixel At parallax Path aggregation cost at that time; For the previous pixel At parallax Path aggregation cost at that time; This is the penalty term applied for a parallax change of 1. For the previous pixel Minimum aggregation cost under all parallaxes; This is the penalty applied for a change in disparity greater than 1.

[0018] This invention introduces an adaptive penalty term related to local structural strength into the cost aggregation formula of dynamic programming. This enables the cost aggregation process to intelligently perceive image content. In edge regions with drastic structural changes, a larger penalty term suppresses sharp jumps in disparity, ensuring smooth and continuous contours. In regions with flat textures, a smaller penalty term allows for minor changes in disparity, thereby protecting the fine structure of the surface. This ensures that the aggregated cost volume can more realistically reflect the depth information of the scene, laying the foundation for generating high-quality disparity maps.

[0019] Preferably, determining the disparity map based on the aggregated cost volume and determining the 3D pose of the target point to be drilled based on the disparity map includes: for each pixel in the aggregated cost volume, selecting the disparity that minimizes the aggregated cost as its optimal disparity to form an initial disparity map; optimizing the initial disparity map to obtain a final disparity map; performing triangulation on the final disparity map based on the camera intrinsic parameter matrix and the binocular baseline length to obtain a 3D point cloud model; extracting the local point cloud of the target point to be drilled and its neighborhood from the 3D point cloud model, calculating the covariance matrix of the local point cloud, and using the eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the surface normal vector of the target point to be drilled; and constructing the 3D pose of the target point to be drilled by combining the 3D coordinates of the target point to be drilled and its surface normal vector.

[0020] After obtaining the disparity map, this invention converts it into a dense three-dimensional point cloud using the principle of triangulation, providing a precise spatial geometric description of the scene. By performing principal component analysis on the local point cloud in the neighborhood of the target point, this invention estimates the surface normal vector of the point, integrates the three-dimensional coordinates and the normal vector to form a complete three-dimensional pose, providing direct and precise execution instructions for the robotic arm controller, ensuring the positioning accuracy and verticality of subsequent operations.

[0021] Secondly, the present invention provides a real-time positioning system for robotic arm operations, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the aforementioned real-time positioning method for robotic arm operations is implemented.

[0022] By adopting the above technical solution, a real-time positioning method for robotic arm operations is generated into a computer program and stored in a memory for loading and execution by a processor. This allows for the creation of a terminal device based on the memory and processor, facilitating its use.

[0023] The beneficial effects of this invention are as follows: By constructing an adaptive aggregation path that closely fits the surface contour of the target object and introducing an adaptive penalty mechanism based on local structural strength, this invention fundamentally solves the limitations of traditional stereo matching algorithms in handling complex geometric structures and varied textured surfaces. It not only improves the accuracy of disparity maps at object edges and irregular curved surface areas, but also enhances robustness to complex lighting environments. This invention can calculate the three-dimensional pose of the target point in real time and accurately, providing reliable visual guidance for high-precision automated operations of robotic arms. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a real-time positioning method for robotic arm operations according to the present invention; Figure 2 The left image is a schematic diagram of the tunnel wall; Figure 3 The image on the right is a schematic diagram of the tunnel wall; Figure 4 Visualize the structural strength weights; Figure 5 This is a visualization of the penalty term P2. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] This invention discloses a real-time positioning method for robotic arm operations, referring to... Figure 1 This includes steps S1-S4: S1. Acquire and preprocess stereo image pairs, and calculate the initial matching cost volume, which consists of the matching cost between pixels in the stereo image pair under different disparities.

[0028] It should be noted that in order to reconstruct the irregular curved 3D model of the tunnel wall and provide a reliable basis for subsequent calculation of borehole normals, the input data for stereo matching must be accurate. Due to inherent distortion in the camera lens and the non-ideal parallelism of the binocular camera installation, model errors are introduced at the source. Therefore, rigorous geometric correction must be performed before stereo matching calculations. Furthermore, tunnels typically rely on a limited number of construction spotlights for illumination, which can create strong local highlights and hard shadows on concrete or rock surfaces. To address this issue, this invention employs a cost calculation method that is insensitive to changes in lighting conditions, ensuring the reliability of initial matching information under complex lighting conditions.

[0029] Specifically, a binocular vision sensor deployed at the end of the robotic arm is used to acquire a pair of stereoscopic images of the tunnel wall in real time, which are used as the left and right images, respectively. For example, Figure 2 The left image is a schematic diagram of the tunnel wall. Figure 3 The image on the right is a schematic diagram of the tunnel wall.

[0030] Using pre-calibrated binocular camera parameters, distortion correction and stereo correction are performed on the left and right images acquired in real time from the end of the robotic arm, so that the epipolar lines of the two images are horizontally collinear, thus obtaining the corrected left and right images.

[0031] Furthermore, to reduce the impact of illumination changes, for each pixel in the corrected left image... Within the preset parallax range Within the image, the matching cost between the image and the corresponding pixel in the right image is calculated. The matching cost is composed of a weighted sum of the Census transform cost and the SAD cost. To ensure consistency of dimensions, the Census transform cost and the SAD cost are normalized before the weighted summation, so that the Census transform cost and the SAD cost are mapped to the interval [0,1].

[0032] Specifically, the matching cost satisfies the expression:

[0033] In the formula, For pixels At parallax The matching cost between the current pixel and the corresponding pixel in the right image; For pixels At parallax The cost of the normalized Census transform of the corresponding pixels in the right image; For pixels At parallax The normalized SAD cost of the corresponding pixel in the right image; These are weighting coefficients. This invention uses weighting coefficients... Balancing the advantages of both costs, the Census transform cost provides robustness to illumination changes, while the SAD cost preserves sensitivity to pixel grayscale details. When the value is high, the matching cost More focused on resisting changes in light; when When the value is low, the matching cost is low. It places more emphasis on the absolute difference in grayscale values.

[0034] The parallax range Based on the robotic arm's working distance and the camera's baseline length, in this embodiment, it is set to [0, 128] pixels. In other embodiments, the implementer can set it according to the actual implementation situation. Weighting coefficient Its function is to balance illumination robustness and detail sensitivity. The value range is [0,1], with an empirical value of 0.8. In other embodiments, the implementer can dynamically adjust it according to the lighting complexity of the actual working environment. The values ​​of are as follows. The Census transform cost and SAD cost are well-known techniques, and their calculation process will not be described in detail here.

[0035] Each pixel in the corrected left image is within the disparity range. Each parallax The matching costs of corresponding pixels in the lower and right images constitute a three-dimensional initial matching cost volume. This three-dimensional initial matching cost volume is a dimensional... A three-dimensional data volume, where each element stores a pixel from the corrected left image. Its corresponding pixel in the right image The matching cost between them, where This indicates the size of the corrected left image. This indicates the row number of the corrected left image. This indicates the column number of the corrected left image.

[0036] S2: Calculate the local structure tensor of each pixel based on one of the stereo image pairs, and generate an adaptive aggregated path field covering the image based on the local structure tensor.

[0037] It should be noted that the mismatch between the cost aggregation path of the traditional SGBM algorithm and the curved surface geometry of the tunnel wall is the core reason for the decrease in positioning accuracy. A fixed linear path will violate the assumption of depth continuity of pixels on the curved surface. An ideal aggregation path should be able to extend along the natural surface contour formed by the tunnel wall in the image. In the image, the direction of these surface contours usually represents the direction of the gentlest local gradient change. The local structure tensor can describe the gradient distribution in the neighborhood. Therefore, this invention analyzes the local structure tensor to find the structural direction that represents the surface contour, thereby providing accurate guidance for generating an adaptive path that fits the geometric features of the scene.

[0038] Specifically, the horizontal gradient of each pixel in the corrected left image is calculated using the Sobel operator. with vertical gradient .

[0039] Furthermore, for each pixel in the corrected left image Calculate pixel points Local structure tensor :

[0040] In the formula, For pixels The local structure tensor; Indicated by A neighborhood window centered on the center; and Each is a pixel in the neighborhood. The horizontal and vertical gradients. The local structure tensor can describe the structural features of a region, effectively distinguishing between structured and unstructured regions in an image, and indicating the orientation of the structure when a pixel... When the local structure tensor is located on the arc-shaped contour formed by the tunnel wall in the image, the gradient directions of the pixels in its neighborhood are highly consistent and perpendicular to the arc-shaped contour. One eigenvalue will be much larger than the other, and their corresponding eigenvectors point to the gradient direction and the contour direction, respectively; conversely, if the pixel... In a flat region, the gradient values ​​of pixels in its neighborhood are small and their directions are random. In this case, the local structure tensor... Both eigenvalues ​​will approach zero. In this embodiment, the neighborhood window The size is 15×15. In other embodiments, implementers can adjust the size of the neighborhood window according to the actual implementation situation.

[0041] Furthermore, for the local structure tensor of each pixel Eigenvalue decomposition yields two eigenvalues. and the corresponding feature vector , will be with smaller eigenvalues corresponding feature vector The local optimal structural direction of this pixel The local optimal structure direction is the direction that extends along the contour in the image. Simultaneously, larger eigenvalues... This represents the structural strength of that pixel.

[0042] Starting from all pixels at the four boundaries of the corrected left image, based on the local optimal structure direction of each pixel... The path extends pixel by pixel into the corrected left image until it reaches the other boundary of the corrected left image, thereby generating a set of aggregated paths that can cover the corrected left image and bend with the content of the corrected left image, forming an adaptive aggregated path field.

[0043] S3. Determine the structural strength of each pixel based on the local structural tensor of each pixel, determine the adaptive penalty term for cost aggregation based on the structural strength, and perform cost aggregation on the initial matching cost volume along each aggregation path in the adaptive aggregation path field using the adaptive penalty term to obtain the aggregated cost volume.

[0044] It should be noted that even if the aggregated path can fit the geometric contour of the tunnel wall, the characteristics of the tunnel wall surface itself are variable. Some areas have clear contours, while others are flat and blurry. Using a fixed smoothing penalty term for cost aggregation will result in insufficient constraints in areas with clear contours, leading to inadequate smoothing, while excessive constraints in flat and blurry areas will result in loss of detail. Therefore, this invention utilizes the structural strength of pixels to dynamically adjust the smoothing penalty term in the dynamic programming formula. This allows the algorithm to apply a larger smoothing penalty in areas with clear contours and defined structures, forcing disparity to remain continuous, while applying a smaller penalty in areas with blurred and uncertain structures, allowing for a certain degree of flexibility in disparity.

[0045] Specifically, the structural strength weight of each pixel is determined based on its structural strength:

[0046] In the formula, For pixels The structural strength weight, with a value range of . ; For pixels The structural strength, i.e., pixel. The maximum eigenvalue of the local structure tensor; To control the coefficient of weight sensitivity, which is used to adjust the response speed of structural strength weight to feature values, this invention uses the mean value of the structural strength of all pixels with a structural strength greater than 0 as... In other embodiments, implementers may set the appropriate parameters according to the actual implementation situation. This invention utilizes an exponential function to distribute characteristic values ​​with an indeterminate range. Mapped to a standard weight range, when pixel When located in a textureless region, Approaching 0, It also approaches 0; when When the point is located in a well-defined, strongly structured region, The value is very large, making Approaching 1.

[0047] For example, Figure 4 Visualize the structural strength weights.

[0048] Based on the structural strength weights of the pixels, an adaptive penalty term is constructed for cost aggregation:

[0049]

[0050] In the formula, and For pixels The adaptive penalty term; Each is a penalty item Minimum and maximum values, penalty terms The minimum and maximum values; For pixels The invention establishes a linear relationship between the penalty value and structural strength to calculate an adaptive penalty term. Its core function is to intelligently adjust the smoothing constraint strength during cost aggregation based on the characteristics of local image regions. In regions with clear structures, such as object edges, the weights are adjusted accordingly. The weights approach 1, causing the penalty term to approach its maximum value, thus imposing a stronger constraint to suppress noise; in flat regions with uniform texture, the weights... Approaching 0 keeps the penalty term near its minimum to avoid over-smoothing and loss of detail; when the structural strength weight When the value changes from 0 to 1, the penalty term also smoothly transitions from the minimum to the maximum value, achieving adaptive adjustment of the penalty intensity.

[0051] Scope of penalties experience value In other embodiments, implementers may adjust the scope of the penalty items according to the actual implementation situation, but it is necessary to meet the following requirements: This is because drastic jumps in parallax should be penalized more severely.

[0052] For example, Figure 5 This is a visualization of the penalty term P2.

[0053] Furthermore, for any aggregated path in the adaptive aggregated path field as the target path, the path aggregation cost of the target path under different disparities at each pixel on the target path is determined: ; In the formula, For the target path at the pixel point Location, parallax Path aggregation cost at that time; For pixels At parallax The matching cost between the current pixel and the corresponding pixel in the right image; For the pixels on the target path The previous pixel; For the previous pixel At the same parallax The aggregation cost below; For the previous pixel At parallax Path aggregation cost at that time; For the previous pixel At parallax Path aggregation cost at that time; This is the penalty term applied for a parallax change of 1. For the previous pixel Minimum aggregation cost under all parallaxes; This is the penalty term applied for disparity changes greater than 1; the third term in the expression. This is used to prevent the path aggregation cost from growing indefinitely during the recursion process, to maintain the validity of the penalty term, and to avoid numerical overflow.

[0054] This invention uses dynamic programming to calculate the minimum cost path from the starting point to the current pixel, which is then used as the path aggregation cost. An adaptive penalty term is introduced during the calculation of the path aggregation cost. and This approach ensures that parallax continuity is maintained in areas with strong structure, while allowing greater parallax variation in areas with weak structure. This effectively balances noise suppression and detail preservation, avoiding insufficient smoothing due to inadequate constraints in areas with clear outlines, and loss of detail due to excessive constraints in areas with flat and blurred surfaces. As a result, the algorithm can provide optimal smoothing constraints when faced with varying surface textures and outlines, enhancing its robustness and environmental adaptability.

[0055] Furthermore, for any pixel in the corrected left image, the path aggregation costs of all aggregation paths passing through that pixel under the same disparity are accumulated and used as the final aggregation cost of that pixel under the corresponding disparity.

[0056] Each pixel in the corrected left image is within the disparity range. Each parallax The final aggregation cost constitutes a three-dimensional aggregation cost volume, wherein the aggregation cost volume is a volume with a dimension of 1. The three-dimensional data volume, in which This indicates the size of the corrected left image. This indicates the row number of the corrected left image. This indicates the column number of the corrected left image.

[0057] S4: Determine the disparity map based on the aggregated cost volume, determine the three-dimensional pose of the target point to be drilled based on the disparity map, and guide the robotic arm drill bit to perform vertical drilling operations.

[0058] It should be noted that after adaptive path aggregation and penalty term constraints, the resulting aggregated cost volume can more accurately reflect the true depth information of the tunnel wall. In order to generate accurate three-dimensional pose commands that can guide the drilling actuator, this invention extracts the optimal disparity from the aggregated cost volume, optimizes the disparity map, and converts the depth information into spatial coordinates and normal vectors.

[0059] Specifically, for each pixel The disparity that minimizes the final aggregation cost is selected as the optimal disparity value for that pixel, and the optimal disparity values ​​of all pixels are used to form the initial disparity map. .

[0060] Furthermore, for the initial disparity map A left-right consistency check is performed to remove unreliable matching points, and the resulting hole regions are filled. Then, noise is smoothed through operations such as weighted median filtering to obtain a high-quality final disparity map. .

[0061] Furthermore, based on the pre-calibrated camera intrinsic parameter matrix and binocular baseline length Using the principle of triangulation, the final disparity map is obtained. Each pixel in and its disparity value Convert to 3D points in camera coordinate system .

[0062] In the obtained 3D point cloud model, the target point to be drilled and its local point cloud within its neighborhood are extracted. Principal component analysis (PCA) is applied to calculate the covariance matrix of this local point cloud, and the eigenvector corresponding to the smallest eigenvalue is determined as the surface normal vector of the target point. It should be noted that the neighborhood size used in PCA defines the local surface region used to calculate the normal. In this invention, the neighborhood used in PCA is a spherical region with a radius of 5 cm centered on the target point to be drilled. In other embodiments, the implementer can adjust the radius of the neighborhood according to the actual implementation situation.

[0063] The target pose, which is composed of the three-dimensional coordinates of the target point to be drilled and its surface normal vector, is sent to the robotic arm motion controller to guide the drill bit to complete high-precision positioning and vertical drilling operations.

[0064] This invention also discloses a real-time positioning system for robotic arm operations, including a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a real-time positioning method for robotic arm operations according to the present invention.

[0065] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

Claims

1. A real-time positioning method for robotic arm operations, characterized in that, include: Obtain the stereo image pair to be processed and calculate the initial matching cost volume, which consists of the matching cost between pixels in the stereo image pair under different disparities; Calculate the local structure tensor of each pixel based on one of the stereo image pairs, and generate an adaptive aggregated path field covering the image based on the local structure tensor. The structural strength of each pixel is determined based on the local structural tensor of each pixel, and an adaptive penalty term for cost aggregation is determined based on the structural strength. Along each aggregation path in the adaptive aggregation path field, the initial matching cost volume is aggregated using the adaptive penalty term to obtain the aggregated cost volume; The disparity map is determined based on the aggregated cost volume, and the three-dimensional pose of the target point to be drilled is determined based on the disparity map, guiding the robotic arm drill bit to perform vertical drilling operations.

2. The real-time positioning method for robotic arm operation according to claim 1, characterized in that, The method for obtaining the matching cost is as follows: Distortion correction and stereo correction are performed on the stereo image pairs; For each pixel in a corrected image, within a preset disparity range, the matching cost between it and the corresponding pixel in another corrected image is calculated. The matching cost is a weighted sum of the Census transform cost and the SAD cost.

3. The real-time positioning method for robotic arm operation according to claim 1, characterized in that, Calculate the local structure tensor of each pixel based on one of the stereo image pairs, including: Calculate the horizontal and vertical gradients of each pixel in the image; for each pixel, construct its local structure tensor based on the horizontal and vertical gradients of each pixel in its neighborhood window.

4. The real-time positioning method for robotic arm operation according to claim 1, characterized in that, Generate an adaptive aggregated path field covering the image based on the local structure tensor, including: Eigenvalue decomposition is performed on the local structure tensor of each pixel to obtain two eigenvalues ​​and the corresponding eigenvectors; the eigenvector corresponding to the smallest of the two eigenvalues ​​is taken as the local optimal structure direction of that pixel. Starting from the boundary pixels of the image, a set of aggregated paths is generated by extending into the image according to the local optimal structure direction of each pixel, thus forming the adaptive aggregated path field.

5. A real-time positioning method for robotic arm operation according to claim 1, characterized in that, Determining the structural strength of each pixel based on its local structure tensor includes: For each pixel, perform eigenvalue decomposition on the local structure tensor, and take the largest eigenvalue among the two decomposed eigenvalues ​​as the structure strength of that pixel.

6. A real-time positioning method for robotic arm operation according to claim 1 or 5, characterized in that, The adaptive penalty term for cost aggregation is determined based on the structural strength, including: Determine the structural strength weight of each pixel based on its structural strength: In the formula, For pixels Structural strength weight; For pixels Structural strength; A coefficient used to control the sensitivity of the weights; It is a natural exponential function; An adaptive penalty term is constructed based on the structural strength weight of each pixel.

7. The real-time positioning method for robotic arm operation according to claim 1, characterized in that, Along each aggregation path in the adaptive aggregation path field, the initial matching cost volume is aggregated using the adaptive penalty term to obtain an aggregated cost volume, including: Determine the path aggregation cost of the aggregation path at each pixel under different parallaxes. Add up the path aggregation costs of all aggregation paths passing through each pixel at the same parallax to get the final aggregation cost of that pixel under the corresponding parallax. Construct an aggregation cost volume from the final aggregation costs of all pixels under all parallaxes.

8. A real-time positioning method for robotic arm operation according to claim 7, characterized in that, The path aggregation cost satisfies the expression: ； In the formula, For the target path at the pixel point Location, parallax Path aggregation cost at that time; For pixels At parallax The matching cost between the current and the corresponding pixel in the right image; For the pixels on the target path The previous pixel; For the previous pixel At the same parallax The aggregation cost below; For the previous pixel At parallax Path aggregation cost at that time; For the previous pixel At parallax Path aggregation cost at that time; This is the penalty term imposed for a parallax change of 1. For the previous pixel Minimum aggregation cost under all parallaxes; This is the penalty applied for a change in disparity greater than 1.

9. A real-time positioning method for robotic arm operation according to claim 1, characterized in that, Determining a disparity map based on the aggregated cost volume, and determining the three-dimensional pose of the target point to be drilled based on the disparity map, includes: For each pixel in the aggregated cost volume, the disparity that minimizes the aggregated cost is selected as its optimal disparity, forming an initial disparity map. The initial disparity map is then optimized to obtain the final disparity map. Based on the camera intrinsic parameter matrix and the binocular baseline length, triangulation is performed on the final disparity map to obtain a 3D point cloud model. In the 3D point cloud model, the target point to be drilled and its local point cloud in its neighborhood are extracted. The covariance matrix of the local point cloud is calculated, and the eigenvector corresponding to the smallest eigenvalue of the covariance matrix is ​​used as the surface normal vector of the target point to be drilled. The 3D coordinates of the target point to be drilled and its surface normal vector constitute the 3D pose of the target point to be drilled.

10. A real-time positioning system for robotic arm operations, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement a real-time positioning method for robotic arm operations according to any one of claims 1-9.

Images