Substation screen cabinet screw identification and positioning method and system based on three-dimensional vision

Abstract

The invention relates to the field of electric power automation, and relates to a transformer substation screen cabinet screw identification and positioning method and system based on three-dimensional vision, and the method comprises the steps: firstly obtaining a conversion relation between a camera coordinate system and a mechanical arm coordinate system, completing hand-eye calibration, shooting screws at multiple positions by a camera, and obtaining corresponding images; obtaining a feature point matching relation between the to-be-processed image and other images through an image feature detection and matching method, and obtaining a point cloud model of the screen cabinet screw; the method comprises the following steps: preprocessing point clouds, segmenting two-dimensional target screw plane point clouds from the preprocessed point clouds, mapping the two-dimensional target screw plane point clouds into three-dimensional point cloud data according to corresponding two-dimensional pixel coordinates, extracting contour points according to the three-dimensional point cloud data corresponding to the model point clouds, and extracting the contour points from scene point clouds, thereby completing point cloud matching and obtaining the pose of a target screw. The invention provides a point cloud registration and splicing algorithm based on image feature auxiliary matching relation generation, and the precision and robustness of a three-dimensional reconstruction algorithm are improved.

Description

Technical Field

[0001] The present invention relates to the field of electric power automation, and in particular to a method and system for identifying and locating screws in a transformer substation panel cabinet based on three-dimensional vision. Background Art

[0002] As the intelligent upgrade of power systems accelerates, the safe operation of substations, as core hub equipment in the power grid, is crucial. Panel cabinets, key components of substation secondary systems, contain hundreds or even thousands of electrical connections secured by screws. The proper assembly of these screws directly impacts the insulation performance and mechanical stability of the equipment. Statistics show that approximately 37% of unplanned substation outages are caused by failed connections within panel cabinets, with over 80% of these incidents resulting from loose, falling, or misaligned screws. Traditional manual inspection methods, plagued by inefficiencies and significant subjective errors, are no longer able to meet the refined O&M requirements of large-scale substation clusters.

[0003] In recent years, automated vision technology has been gradually introduced into the industrial inspection field to replace manual visual inspection. While existing screw detection solutions based on two-dimensional image processing have certain practical applications, they face multiple technical bottlenecks in actual application. Few solutions are specifically designed for the identification and positioning of screws in panel cabinets. The main problems include: First, the internal spatial structure of panel cabinets is complex, and visual occlusion caused by adjacent components makes it difficult to extract two-dimensional image features; second, image information from a single plane cannot effectively represent the three-dimensional spatial position of the screws, making it difficult to determine whether they meet millimeter-level installation accuracy requirements; and third, in environments with strong electromagnetic interference, the robustness and repeatability of traditional vision systems are significantly reduced. The above-mentioned shortcomings make it difficult for existing technologies to achieve full lifecycle monitoring and management of the status of screws in substation panel cabinets.

[0004] For example, the invention patent with publication number CN119919600A discloses a method for constructing a three-dimensional map of a substation based on point cloud registration. It uses an iterative closest point (ICP) algorithm to generate pseudo-labels for unregistered point clouds, and combines them with a small number of real labels through deep neural network training and optimization, gradually improving the registration accuracy of the point cloud and the efficiency of three-dimensional map construction. The present invention can effectively solve the problem of scarcity of real label data caused by equipment occlusion in substation scenarios, significantly improve the construction quality and efficiency of three-dimensional electronic maps of substations, and has important practical application value. The invention patent with publication number CN 118628569A discloses a method for positioning screw hole posture based on 3D point cloud. First, the object is measured in 2D and 3D, the screw hole is positioned in the 2D map, the 3D point cloud is cut according to the 2D positioning screw hole, and then the cut point cloud is gridded. A circle is fitted around the circular hole according to the grid, a circular plane is fitted according to the fitted circle, and then the point cloud is fitted according to the plane. Then, the screw hole fitting point cloud is template matched according to the screw hole model to determine the screw hole posture. That is, the present invention calculates the hole posture based on the hole plane fitting of the point cloud edge. Compared with the prior art in which 3D object reconstruction is achieved through posture relationship and captured image features, the influence of captured image quality on 3D modeling is reduced, and the operation is simpler, faster and more accurate. However, it does not consider the problem of noise points and the positioning effect cannot be guaranteed. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a substation panel cabinet screw identification and positioning method based on three-dimensional vision. This method solves the problem of inaccurate identification and positioning of panel cabinet screws. The present invention also provides a substation panel cabinet screw identification and positioning system based on three-dimensional vision.

[0006] Technical solution: According to a first aspect of the present invention, a method for identifying and locating screws in a substation cabinet based on three-dimensional vision is provided, the method comprising:

[0007] Obtaining the conversion relationship between the camera coordinate system and the robotic arm coordinate system to complete the hand-eye calibration, wherein the camera captures the screws in multiple positions to obtain corresponding images;

[0008] Normalize the brightness of the image to be processed, perform a differential operation between the background image and the current normalized image to be processed, and remove the invalid area in the differential image to obtain the area of ​​interest, that is, the cabinet screw area;

[0009] The feature point matching relationship between the image to be processed and other images is obtained through image feature detection and matching methods, thereby obtaining the corresponding point cloud data;

[0010] The surface point cloud information of the screen cabinet screw is obtained based on the screen cabinet screw area, the feature point matching relationship and the point cloud data. The point cloud correspondence is screened by a random sampling consistency method. The point cloud pose transformation matrix is ​​obtained based on the screened point cloud correspondence. The optimal solution of the point cloud pose transformation matrix is ​​solved. The two frames of point clouds are spliced ​​based on the obtained optimal solution. This step is repeated to obtain a point cloud model of the screen cabinet screw.

[0011] The straight-through filtering and statistical filtering algorithms are used to remove irrelevant point clouds from the model point cloud. The improved voxel filtering method is used to downsample the point cloud data. The improved radius filtering method is then applied to remove noise point clouds. Finally, the pre-processed point cloud of the cabinet screw end face is extracted.

[0012] Segment a two-dimensional target screw plane point cloud from the preprocessed point cloud, map it to three-dimensional point cloud data according to its corresponding two-dimensional pixel coordinates, extract contour points from the three-dimensional point cloud data corresponding to the model point cloud, and extract contour points from the scene point cloud, thereby completing point cloud matching and obtaining the position and pose of the target screw;

[0013] The position of the target screw is converted to the robotic arm coordinate system through the hand-eye matrix, and the robotic arm is controlled to operate the target screw.

[0014] Further, including:

[0015] The brightness normalization of the image to be processed includes:

[0016] Calculate the average grayscale value of each frame in the first N frames, and record the image corresponding to the median of the average grayscale value as M;

[0017] Calculate the average pixel value μ of image M m and pixel value standard deviation σ m And the average pixel value μ of the image P to be processed p and pixel value standard deviation σ p ;

[0018] Thus, the brightness normalization coefficient α is obtained, and the calculation formula is as follows:

[0019] min((αμ m -μ p ) 2 +(α 2 σ m -σ p ) 2 ),α∈[0,1];

[0020] Where, (αμ m -μ p ) 2 Represents the difference between the average pixel values ​​of M and P, (α2 σ m -σ p ) 2 Represents the difference between the standard deviations of the pixel values ​​of M and P.

[0021] Multiply all pixel values ​​of image P by the brightness normalization coefficient α so that the average brightness of M and P are close and the brightness distribution is similar, thereby completing image normalization.

[0022] Further, including:

[0023] The step of performing a difference operation on the background image and the currently normalized image to be processed and removing invalid areas in the difference image includes:

[0024] Traverse the differential image, and for the current pixel point, if it has been marked as belonging to a connected area, skip it; otherwise, use the current point as the seed to start growing the connected area. Through the breadth-first search algorithm, all other pixel points connected to the seed point are found, marked with a new area number, and the number of pixels in the current area is counted. The connected area formed is traversed, and the point with the largest number of pixel values ​​is taken as the area of ​​interest, that is, the screen cabinet screw area.

[0025] Further, including:

[0026] The method of obtaining a feature point matching relationship between the image to be processed and other images by an image feature detection and matching method, thereby obtaining corresponding point cloud data, includes:

[0027] Since each point in the point cloud is generated by a pixel in the grayscale image, each point in the point cloud can find a corresponding pixel in the grayscale image. Based on this one-to-one correspondence, the matching relationship between the pixels in the two grayscale images is obtained and then mapped to the point cloud, and the corresponding relationship between the points in the two point clouds is obtained, thereby obtaining the corresponding point cloud data between the point clouds.

[0028] Further, including:

[0029] Obtaining a point cloud pose transformation matrix based on the filtered point cloud correspondence relationship and solving the optimal solution of the point cloud pose transformation matrix includes:

[0030] The objective function is solved by a nonlinear optimization algorithm based on Levenberg-Marquardt, thereby obtaining the optimal solution of the point cloud pose transformation matrix.

[0031] Further, including:

[0032] The method of removing irrelevant point clouds from the model point cloud by using the straight-through filtering and statistical filtering algorithms includes:

[0033] In the cabinet screw work scenario, a bounding box is constructed and the maximum and minimum values ​​of the x, y, and z coordinates are set. Based on the set threshold range, data within the specified range is filtered and retained, and data outside the range is treated as outliers and removed. Statistical filtering algorithms are used to reduce high-frequency noise in point cloud data.

[0034] Further, including:

[0035] The method of reducing high-frequency noise in point cloud data by using a statistical filtering algorithm includes:

[0036] For each point p in the model point cloud i , determine its neighborhood N, the neighborhood N is based on point p i A sphere of fixed size centered at i The average distance d of points in the neighborhood mean and standard deviation d std , respectively:

[0037]

[0038] Judgment point p i Is it an outlier: For each point, if the distance d from the point in the neighborhood to it is p If the difference between the average distance and the point exceeds a certain threshold, it will be marked as an outlier. The judgment condition is: |d p -d mean |>k*d std , k is the definition threshold;

[0039] Remove outliers: Remove or replace points marked as outliers from the point cloud to obtain a new point cloud that does not contain outliers.

[0040] Further, including:

[0041] The improved voxel filtering method is used to complete the downsampling processing of the point cloud data, including:

[0042] Set the voxel size corresponding to the new point cloud data to N; divide the point cloud data into sub-regions with voxels as the unit, and each voxel is regarded as a cube or cube grid unit;

[0043] For each voxel, calculate the centroid of all points in it, that is, the average value of the coordinates, and let the point set in the voxel be P v ={p1,p2,…,p i}, where p i is the coordinate of the i-th point, then the centroid p c The calculation formula is:

[0044]

[0045] The entire point cloud constructs a KD tree to quickly find the point closest to the centroid; for each voxel, the distance to the centroid p is found through the KD tree. c The nearest point p cr ; Use the nearest point p cr Replace all point clouds in a voxel, that is, all points p in a voxel i Replace with the nearest point p cr ;

[0046] Repeat the above steps for all voxels until all voxels have been processed.

[0047] Further, including:

[0048] The improved radius filtering method is applied to remove noise point clouds and finally extract the pre-processed point clouds of the screen cabinet screw end faces, including:

[0049] Assume that the input point cloud data set is Q, the filter radius is d, the quantity threshold is k, and Q is expressed as follows:

[0050] Q={q1,q2,…,q n};

[0051] Select any point q in Q i As the search seed point, with d as the search radius, count the number of points m in the neighborhood. If the relationship between m and k meets the requirements, it will be retained as q′ i , m is represented as follows:

[0052]

[0053] In the formula, B(q i ,r) indicates q i is the center of the sphere and r is the number of data points in the spherical space with radius;

[0054] Repeat the above steps to traverse each point in the set Q, and reorganize all points that meet the requirements into a point cloud set Q′ i .

[0055] Q′ i ={q′1,q′2,…,q j}={q′ i ∈Q|B(q′ i ,r)>k}.

[0056] Further, including:

[0057] The extracting of contour points according to the three-dimensional point cloud data corresponding to the model point cloud comprises:

[0058] First, calculate the normal vector of each point in the model point cloud, through pi and its nearest neighbor set N i The mean of all points in Calculate p i The covariance matrix M i , expressed as:

[0059]

[0060] Where n is p i The number of points in the field is M i Perform eigenvalue decomposition to obtain eigenvectors and eigenvalues;

[0061]

[0062] Where, is the eigenvector, is the characteristic value, let Then the minimum eigenvalue λ is the point p i The normal vector of

[0063] Then find the adjacent points of each point, and then calculate the angle between the normal vector of each point and the normal vector of its adjacent points. Points whose angle exceeds a preset threshold are marked as contour points.

[0064] Further, including:

[0065] The method of extracting contour points from the scene point cloud to complete point cloud matching and obtain the position and posture of the target screw includes: rough matching and fine matching of point-to-point features of contour points. The fine matching of point-to-point features of contour points includes:

[0066] Initialization: Obtain source point cloud P and target point cloud Q through coarse matching, initial transformation T0 = (R0, t0), set acceleration parameters, history window size m, convergence threshold ε, maximum number of iterations I max ;

[0067] Iterative optimization repeats the following steps until convergence or the maximum number of iterations is reached:

[0068] For the current source point cloud P k Apply transformation T k , get the aligned point cloud P′ k ; Calculate the error function E(Δ) of the corresponding point pair and solve the increment Δ that minimizes the error (k) ; Maintain a historical transformation increment queue H = [Δ (0) ,Δ (1) ,…,Δ (k) ], the length does not exceed m; when the queue length exceeds m, remove the earliest increment;

[0069] Construct matrix A and vector b based on historical increments to solve the optimal weight β i :

[0070]

[0071] Solve β=(A T A) -1 A T b;

[0072] Calculate the weighted combination:

[0073]

[0074] Update the transformation based on the predicted increment:

[0075] T k+1 =T k +Δ pred

[0076] Apply the new transformation T to the source point cloud P k+1 , and get P′ k+1 ;

[0077] Convergence judgment: Calculate the current error RMSE. If the RMSE change is less than the threshold ε or the maximum number of iterations is reached, the iteration is terminated.

[0078] On the other hand, the present invention also provides a substation cabinet screw identification and positioning system based on three-dimensional vision, the system comprising:

[0079] An image acquisition module is used to obtain the conversion relationship between the camera coordinate system and the robotic arm coordinate system to complete hand-eye calibration. The camera captures screws in multiple positions to obtain corresponding images.

[0080] The effective area calculation module is used to normalize the brightness of the image to be processed, perform a differential operation between the background image and the current normalized image to be processed, and remove the invalid area in the differential image, so as to obtain the area of ​​interest, that is, the screen cabinet screw area, in the differential image;

[0081] The point cloud formation module is used to obtain the feature point matching relationship between the image to be processed and other images through image feature detection and matching methods, thereby obtaining the corresponding point cloud data;

[0082] A point cloud model construction module is used to obtain surface point cloud information of the screen cabinet screw based on the screen cabinet screw area, feature point matching relationship and point cloud data, screen the point cloud correspondence relationship through a random sampling consistency method, and obtain a point cloud pose transformation matrix based on the screened point cloud correspondence relationship, solve the optimal solution of the point cloud pose transformation matrix, and splice two frames of point clouds based on the obtained optimal solution; repeat this step to obtain a point cloud model of the screen cabinet screw;

[0083] The preprocessing module is used to remove irrelevant point clouds from the model point cloud using straight-through filtering and statistical filtering algorithms, downsample the point cloud data using an improved voxel filtering method, and then remove noise point clouds using an improved radius filtering method. Finally, the preprocessed point cloud of the cabinet screw end face is extracted.

[0084] The target screw pose calculation module is used to segment the two-dimensional target screw plane point cloud from the preprocessed point cloud, map its corresponding two-dimensional pixel coordinates to the three-dimensional point cloud data, extract contour points based on the three-dimensional point cloud data corresponding to the model point cloud, and propose contour points from the scene point cloud, thereby completing point cloud matching and obtaining the pose of the target screw;

[0085] The control module is used to convert the position of the target screw into the robotic arm coordinate system through the hand-eye matrix and control the robotic arm to operate the target screw.

[0086] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0087] (1) In the structured light 3D reconstruction part, this paper proposes a point cloud registration and stitching algorithm based on image feature-assisted matching relationship generation. To address the structural characteristics of cabinet screws and the shortcomings of traditional point cloud registration algorithms, the matching relationship between point clouds is obtained through image feature detection and matching. To address the problem of many mismatches in the matching relationship, the correct matching relationship is screened out by extracting the cabinet screw area in the image and using a random sampling consistency algorithm, thereby improving the accuracy and robustness of the algorithm.

[0088] (2) The present invention proposes a neighbor search-based voxel filtering method for point cloud preprocessing. In practical applications, the center of gravity of individual voxels may not be the actual measured point. This approximate substitution, in a sense, increases the measurement error for subsequent screw tightening operations. To address this, the present invention makes a slight improvement to the voxel point extraction rule. After calculating the center of gravity, a neighbor search is performed on it, selecting the nearest point as the voxel point.

[0089] (3) The present invention proposes an improved radius filtering method with a minimum number of points constraint in the point cloud preprocessing part. In the traditional radius filtering algorithm, the neighbor search for each query point will traverse all points within the specified radius. After the traversal is completed, the number of search points is counted and compared with the set threshold to determine whether the query point is a noise point. Such a calculation process is huge in terms of time and space complexity. In order to improve the efficiency of the algorithm, the present invention improves the traversal rule based on the traditional algorithm: set the maximum traversal number n, whose value is equal to the radius filter setting threshold k. When performing a neighborhood search on a query point, there is no need to traverse all points within the search radius.

[0090] (4) In the point cloud segmentation part, the present invention proposes a point cloud segmentation algorithm based on the mapping relationship between two-dimensional images and three-dimensional point clouds, processes point cloud data in stages, reduces the dimensionality of three-dimensional data, and significantly improves the efficiency of point cloud segmentation.

[0091] (5) The present invention adopts a point pair feature matching algorithm based on point cloud contour points and introduces curvature features to solve the problem of point pairs with repeated features.

[0092] (6) The present invention adopts a multi-scale point pair feature representation method: only the point cloud normal and the line relationship between the point pairs are used as feature representation, while taking into account the curvature information. This feature representation can better distinguish different types of workpieces. In addition, the point pair features have good invariance and distinguishability, and can maintain consistency even in different postures and viewing angles. Therefore, even in the presence of large initial posture deviations, these feature descriptors can still provide reliable matching information, which helps the algorithm find the correct correspondence. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a flow chart of a method for identifying and locating screws in a substation cabinet based on three-dimensional vision, proposed in an embodiment of the present invention;

[0094] Figure 2 This is a flow chart of a point cloud registration and stitching algorithm based on image feature-assisted matching relationship generation according to an embodiment of the present invention;

[0095] Figure 3 is a flow chart of the improved background difference method according to an embodiment of the present invention;

[0096] Figure 4 This is a flowchart of the screen cabinet screw point cloud segmentation described in an embodiment of the present invention. DETAILED DESCRIPTION

[0097] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention and not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0098] Example 1: The embodiment of the present invention discloses a method for identifying and locating screws in a substation cabinet based on three-dimensional vision, see Figure 1 , the method comprises the following steps:

[0099] Step 1: Stereo and hand-eye calibration. Specifically, the system first performs checkerboard calibration to obtain the camera's internal and external parameters. The projector is considered the right camera, and the camera corners are mapped to the projector using a homography matrix to complete stereo calibration. A calibration plate is then placed on the robotic arm, and the teach pendant is used to control the robotic arm. The Zhang calibration method is used to obtain the transformation relationship between the camera coordinate system and the robotic arm coordinate system to complete hand-eye calibration.

[0100] System calibration, including stereo calibration and hand-eye calibration, involves the following steps: First, the intrinsic and extrinsic parameters of the camera and projector must be obtained. Camera calibration can obtain these parameters. Since cameras and projectors share similar optical properties, similar calibration methods can also be used to obtain the intrinsic and extrinsic parameters of the projector. Next, the relationship between the camera and projector is determined through coordinate system transformation. By transforming the coordinate systems of the camera and projector, the exact position and orientation relationship between them can be determined, thereby enabling coordinate conversion between two-dimensional images and three-dimensional space. The camera in this system is mounted above the robotic arm, forming an eye-out-of-hand system. Typically, in an eye-out-of-hand system, the robotic arm moves a calibration plate to different positions and captures images of the plate in different poses. The camera's extrinsic parameters relative to the plate are then calculated. Combined with the robotic arm's motion pose parameters stored in the teach pendant, the positional relationship between the camera and the end of the robotic arm, known as the hand-eye matrix, is then determined.

[0101] Step 2: Based on the point cloud registration and stitching algorithm generated by the image feature-assisted matching relationship, three-dimensional reconstruction is performed to generate a point cloud. The specific steps include: data acquisition, point cloud generation, point cloud registration and stitching.

[0102] This paper proposes a point cloud registration and stitching algorithm based on image feature-assisted matching relationship generation. In view of the structural characteristics of cabinet screws and the shortcomings of traditional iterative closest point algorithms, the matching relationship between point clouds is obtained through image feature detection and matching. In order to solve the problem of many mismatches in the matching relationship, the correct matching relationship is screened out by extracting the cabinet screw area in the image and the random sampling consistency algorithm, thereby improving the accuracy and robustness of the algorithm. The algorithm flow is as follows Figure 2 As shown, the specific steps are:

[0103] a) Data acquisition: A binocular structured light camera captures the screw workpiece at multiple locations and saves its point cloud data;

[0104] b) Region of interest extraction: Taking the grayscale image as input, the screen cabinet screw area in the grayscale image is extracted as the region of interest through the improved background difference method. In order to distinguish the static background from the screen cabinet screws, the screen cabinet screw area in the grayscale image is extracted as the region of interest, and only the region of interest is considered in image feature detection and matching to ensure that the accurate screen cabinet screw pose transformation matrix can be obtained in the point cloud registration.

[0105] In order to enhance the robustness of the background difference algorithm, the present invention improves the traditional background difference method. First, the brightness of the image is normalized to eliminate the influence of the overall illumination change. Then, the background is updated by the sliding window method to avoid the background changes caused by the cabinet wiring harness. Finally, the image expansion algorithm is used to fill the holes and the image connectivity criterion is used to remove the invalid areas.

[0106] The algorithm applies a Gaussian filter to the grayscale image before performing a difference to reduce random noise. To remove invalid regions from the differenced image, the algorithm uses an invalid region removal method based on the number of pixels in each region. This method determines the region size by calculating the number of connected components and pixels in each region, then removes small regions.

[0107] The improved background difference method process is as follows Figure 3 As shown. The present invention performs Gaussian filtering on the grayscale image and then performs differential processing to reduce random noise. Gaussian filtering can smooth and remove noise from the image, but it will also make the processed image more "blurry". The present invention performs brightness normalization on the grayscale image to eliminate the influence of ambient light. The brightness normalization steps are as follows:

[0108] Step 1: Calculate the average grayscale value of each frame in the first N frames, and record the image corresponding to the median of the average grayscale value as M;

[0109] Step 2: Calculate the average pixel value μ of M m and pixel value standard deviation σ m ;

[0110] Step 3: Let the current image to be processed be P, and calculate the average pixel value μ of P p and pixel value standard deviation σ p ;

[0111] Step 4: Calculate the brightness normalization coefficient α. The calculation formula is as follows:

[0112] min((αμ m -μp ) 2 +(α 2 σ m -σ p ) 2 ),α∈[0,1](1)

[0113] In the formula, by solving the min() function, we can get the value of α when the expression value is the smallest, (αμ m -μ p ) 2 Represents the difference between the average pixel values ​​of M and P, (α 2 σ m -σ p ) 2 Represents the difference between the standard deviations of the pixel values ​​of M and P.

[0114] After obtaining the brightness normalization coefficient α, all pixel values ​​of P are multiplied by the brightness normalization coefficient α so that the average brightness of M and P is close and the brightness distribution is similar.

[0115] The present invention uses a camera to capture the initial background. After the background image is updated, a difference operation is performed between the background image and the current image, and the difference result is then binarized to obtain a difference image. The large white area in the center of the difference image is the region of interest, namely the cabinet screw area. However, outside the region of interest, there are still a large number of invalid areas in the difference image that need to be further removed. In addition, there are also some holes within the region of interest that need to be filled.

[0116] To remove invalid regions from the difference image, the present invention employs an invalid region removal method based on the number of pixels in each region. This method determines the region size by calculating the connected domain and the number of pixels in each region, then removes small regions. The pixel adjacency of the connected domain plays a key role in this process. The present invention employs a four-adjacency relationship and a seed-filling method to obtain connected regions from the difference binary image.

[0117] The specific implementation method is to traverse the image. If the current pixel is already marked as belonging to a connected region, it is skipped; otherwise, the current pixel is used as a seed to start growing the connected region. Using the breadth-first search algorithm, all other pixels connected to the seed point are found, marked with a new region number, and the number of pixels in the current region is counted. The steps of the algorithm for removing invalid regions are as follows (the input is a binary image F, and the output is the image of interest):

[0118] Initialization: Select 4 adjacencies, initialize the stack, and initialize the region size N = 0.

[0119] Step 1: Select an unmarked point p in F as a seed and create a new connected region K;

[0120] Step 2: Set the mark of point p to K, N = N + 1;

[0121] Step 3: Traverse the four adjacent points of p and push the white points into the stack;

[0122] Step 4: Check if the stack is empty. If not, pop the top pixel of the stack as the new point p and return to step 2. If so, all points in the current connected region are marked as K, and the size of this connected region is N.

[0123] Step 5: Determine whether there are unmarked points in F. If so, set N = 0 and return to step 1. Step 6: Traverse the connected area and take the point with the largest number of pixel values ​​as the region of interest. The algorithm ends.

[0124] c): Image feature detection and matching: With grayscale images as input, the feature point matching relationship between grayscale images is obtained through image feature detection and matching algorithms. Since point clouds only contain three-dimensional spatial position information, feature detection can only extract features through information such as curvature changes and point cloud density. It is easy to detect a large number of similar feature points, resulting in a large number of erroneous correspondences in the matching results. The point cloud model generated by the three-dimensional reconstruction part is the basis for a series of subsequent operations such as preprocessing and matching. The point clouds involved in operations such as point cloud preprocessing and segmentation are all based on model point clouds, in order to obtain an accurate model as a benchmark for subsequent matching.

[0125] To address this issue, the 3D reconstruction of cabinet screws employs feature detection and matching on grayscale images. The corresponding point sets between the grayscale images and point clouds are then derived from the correspondence between them. This is because each point in a point cloud is generated from a pixel in the grayscale image. Therefore, each point in the point cloud can be mapped to a corresponding pixel in the grayscale image. Based on this one-to-one correspondence, as long as the matching relationship between the pixels in the two grayscale images can be established, they can be mapped to the point clouds, thereby obtaining the corresponding relationship between the points in the two point clouds. This method effectively improves the accuracy and stability of point cloud registration.

[0126] This embodiment does not limit the specific feature detection and matching algorithm. For example, the SIFT algorithm or the ORB algorithm can be used as the basis of the matching algorithm of this application.

[0127] Cabinet screws are typically made of aluminum alloy, carbon steel, or stainless steel. However, these materials have a monochromatic surface color and lack texture. This leads to frequent mismatches during image feature detection and matching, making them unsuitable for subsequent point cloud correspondence generation.

[0128] To address this issue, the present invention improves upon the image feature detection and matching algorithm employed by the application by designing a random square mosaic pattern as a visual label, consisting of 400 randomly colored squares. This visual label is robust to translation, rotation, scaling, and projection transformations.

[0129] d) Screen and cabinet screw surface point cloud information generation and point cloud stitching: Using the screen and cabinet screw area, feature point matching relationship, and point cloud as input, the point cloud pose transformation matrix is ​​obtained through three steps: point cloud matching relationship generation, establishing a pose transformation objective function, and solving a nonlinear optimization algorithm. Based on 2D image feature matching, the point cloud correspondence is directly obtained based on the one-to-one correspondence between the grayscale image and the point cloud. After obtaining the point cloud correspondence, the rigid body pose transformation principle is used for point cloud registration:

[0130] First, the point cloud correspondence is screened through the random sampling consistency algorithm to reduce the point cloud registration error caused by the point cloud correspondence error; then, the point cloud pose transformation matrix is ​​established according to the point cloud correspondence to solve the objective function;

[0131] Then, the objective function is solved by the nonlinear optimization algorithm based on Levenberg-Marquardt to obtain the optimal solution of the point cloud pose transformation matrix;

[0132] Finally, the point cloud is transformed and then stitched together to complete the registration of the two frames of point cloud. By repeating the point cloud registration and point cloud stitching process until all the data is processed, the point cloud model of the cabinet screw is obtained.

[0133] Step 3: Preprocess the point cloud to make it smoother and remove noise and outliers. The specific steps are: straight-through filtering, statistical filtering, improved voxel filtering with nearest neighbor search, and improved radius filtering with minimum point count constraint.

[0134] The point cloud preprocessing steps are as follows: first, the straight-through filtering and statistical filtering algorithms are applied to remove most of the irrelevant point clouds in the original point cloud. Secondly, voxel filtering is used to complete the downsampling processing of the point cloud data. Finally, radius filtering is applied to complete the removal of noise point clouds. Finally, the point cloud of the screen cabinet screw end face is extracted to complete the point cloud preprocessing.

[0135] The steps of through-filtering are as follows: In the screen cabinet screw working scene, the preprocessed object is a three-dimensional point cloud, so the through-filtering constructs a bounding box to select the approximate location of the model, reducing the range that needs to be identified and improving efficiency. First, a bounding box is constructed and the maximum and minimum values ​​of the x, y, and z coordinates are set; based on the set threshold range, the data within the specified range is filtered and retained, and the data outside the range is regarded as an outlier and eliminated.

[0136] Statistical filtering algorithms are used to smooth point cloud data and reduce high-frequency noise. By calculating the statistical information of points in a neighborhood and replacing the current point, noise can be suppressed, making the point cloud more readable and visually effective.

[0137] The steps are:

[0138] a) For each point p i , determine its neighborhood N. The neighborhood is based on point p i A sphere of fixed size centered on

[0139] b) Calculate each point p i The average distance d of points in the neighborhood mean and standard deviation d std , respectively:

[0140]

[0141] c) Determine point p i Is it an outlier? For each point, if the distance d from the point in the neighborhood to it is p If the difference between the average distance and the point exceeds a certain threshold (set as a multiple of the standard deviation), it will be marked as an outlier. The judgment condition is: |d p -d mean |>k*d std , k is the definition threshold.

[0142] d) Remove outliers: Remove or replace the points marked as outliers from the point cloud to obtain a new point cloud that does not contain outliers.

[0143] To improve the efficiency of subsequent point cloud matching, this paper proposes a voxel filtering method using a nearest neighbor search. In practical applications, the center of gravity of individual voxels may not be the actual measured point. This approximate substitution, in a sense, increases measurement errors during subsequent screw tightening operations. To address this, this paper makes a slight improvement to the voxel point extraction rule. After calculating the center of gravity, a nearest neighbor search is performed on it, selecting the nearest point as the voxel point. The steps are as follows:

[0144] Initialization: Given point cloud data, set the voxel size to N.

[0145] Step 1: Voxelization: Divide the point cloud data into sub-regions in units of voxels. Each voxel can be regarded as a cube or a cubic grid unit.

[0146] Step 2: Find the centroid: For each voxel, calculate the centroid of all points in it, that is, the average value of the coordinates. Let the set of points in the voxel be P v ={p1,p2,…,p i}, where p iis the coordinate of the i-th point, then the centroid p c The calculation formula is:

[0147]

[0148] Step 3: Construct KD tree: Construct KD tree for the entire point cloud to quickly find the point closest to the centroid. Find the nearest point: For each voxel, find the distance to the centroid p through the KD tree c The nearest point p cr ;

[0149] A KD tree (K-Dimensional Tree) is a binary tree data structure used to efficiently organize k-dimensional spatial data. It is primarily used to accelerate operations such as nearest neighbor searches and range searches. It recursively partitions the space and organizes data points into a tree structure, significantly reducing the amount of computation required during searches.

[0150] Step 4: Replace: Use the nearest point p cr Replace all point clouds in the voxel. That is, all points p in the voxel i Replace with the nearest point p cr ;

[0151] Step 5: Iteration: Repeat steps 2 to 4 for all voxels until all voxels are processed. This downsampling method is very useful in real-time processing and accelerating point cloud data processing tasks.

[0152] Improved radius filtering with minimum point count constraint. In the traditional radius filtering algorithm, the nearest neighbor search for each query point will traverse all points within the specified radius. After the traversal is completed, the number of search points is counted and compared with the set threshold to determine whether the query point is a noise point.

[0153] This computational process is enormous in terms of both time and space complexity. To improve algorithm efficiency, this paper improves on the traditional algorithm's traversal rules by setting a maximum traversal number n, whose value is equal to the radius filter threshold k. When searching the neighborhood of a query point, it is not necessary to traverse all points within the search radius. This improved algorithm is significantly more efficient than traditional radius filtering. The specific steps of the algorithm are as follows:

[0154] Step 1: Let the input point cloud set be Q, the filter radius be d, the number threshold be k, and Q be as follows:

[0155] Q={q1,q2,…,q n} (5)

[0156] Step 2: Select any point q in Q iAs the search seed point, with d as the search radius, count the number of points m in the neighborhood. If the relationship between m and k meets the requirements, it will be retained as q′ i , m is as follows:

[0157]

[0158] In the formula, B(q i ,r) indicates q i is the center of the sphere and r is the number of data points in the spherical space with a radius.

[0159] Step 3: Repeat step 2 to traverse each point in the set Q, and reorganize all points that meet the requirements into the point cloud set Q′ i .

[0160] Q′ i ={q′1,q′2,…,q j}={q′ i ∈Q|B(q′ i ,r)>k} (7)

[0161] Step 4: Based on the point cloud segmentation algorithm of the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the target screw plane point cloud is segmented from the preprocessed point cloud. The specific steps include: threshold segmentation, hole filling, morphological opening operation, connected domain analysis and combining the mapping relationship to achieve the segmentation of the workpiece point cloud.

[0162] Point cloud segmentation algorithm based on the mapping relationship between two-dimensional image and three-dimensional point cloud. The algorithm flow is as follows Figure 4 As shown in the figure, the model screw plane point cloud is segmented from the preprocessed point cloud. The specific steps include threshold segmentation, hole filling, morphological opening operation, connected domain analysis, and combining mapping relationships to achieve workpiece point cloud segmentation. Compared with traditional segmentation algorithms, this algorithm achieves similar segmentation results but significantly reduces segmentation time.

[0163] The computational complexity of three-dimensional point cloud data is too large, the computational speed is slow, and the requirements for hardware such as GPUs are higher. If only a small number of key points in the data are selected for calculation, the results may be inaccurate due to too little information, while selecting too many key points will cause the computational speed to be too slow. The algorithm proposed in this invention adopts a two-stage approach to achieve the pose estimation of the target workpiece. The first stage is based on two-dimensional image processing, and uses digital image technology to segment and extract the local point cloud area containing the point cloud data of a single workpiece, eliminate useless background point clouds, and reduce the computational complexity; the second stage is based on the matching of three-dimensional local area point cloud data to achieve the pose estimation of the target workpiece.

[0164] The steps of 2D image segmentation include: height information extraction, background separation, hole filling, noise point removal, connected domain extraction, and 2D coordinate extraction. After the 2D coordinates are extracted, they need to be converted into the corresponding 3D point cloud data, that is, to achieve the conversion of 2D pixel coordinates into 3D spatial coordinates.

[0165] Background separation: Background separation uses an overfitting plane approach. A plane fit is performed on the captured scene image data to obtain depth data for the plane where the cabinet is located. The two depth values ​​are subtracted, and the grayscale values ​​of the resulting image represent the height of the cabinet screws in the scene. A global threshold segmentation approach is then used to extract the target area in the height image. The global threshold is selected based on the grayscale distribution of the height image.

[0166] Hole filling: After the target workpiece is separated from the background by plane fitting and threshold segmentation, some screw workpiece images have hole areas with pixel grayscale values ​​of 0. The holes are formed because the height of the workpiece in this area is relatively small and close to the background height of the screen cabinet. The height difference map is locally lower than the segmentation threshold. The segmentation process defines it as background removal, resulting in missing pixel data in this area. Therefore, in order to ensure the integrity of the data, it is necessary to fill the holes in the workpiece image with holes. The present invention fills the holes in the image with the average grayscale value of the pixels in the adjacent window of its pixel points, so as to maintain the accuracy of the data without changing the grayscale value of the surrounding pixels while filling the hole area.

[0167] Noise point removal: After hole filling, some noise points may still remain in the image due to environmental noise and mixed pixels. These points are either scattered far from the workpiece area or adhere to the workpiece area. This can affect workpiece image segmentation and subsequent workpiece point cloud matching, reducing the accuracy of the final pose estimation. This method uses a morphological opening operation to remove these noise points. The structuring element is a circle with a fixed radius parameter, which effectively removes noise points from the image.

[0168] Connected Domain Extraction: After 2D image processing, including background separation, hole filling, and noise removal, the workpiece image is roughly separated into distinct regions, thus meeting the prerequisites for segmentation and extraction. This study employs image connected domain analysis to segment the workpiece. To prevent interference from previously processed data that cannot be removed, the extracted connected domains are screened by area, eliminating areas with excessively small areas.

[0169] After the workpieces are segmented, their corresponding 2D coordinates need to be obtained to convert them into 3D point cloud data. After connected domain analysis, different connected domains in the image are labeled with distinct digital labels, meaning each screw workpiece region has a corresponding digital label. Therefore, the digital labels make it easy to find and extract the regions where different workpieces are located, along with their 2D pixel coordinates.

[0170] 2D and 3D data mapping: Pixel coordinates and spatial coordinates have a definite mapping relationship. After 2D image processing, the 2D coordinates of each screw workpiece area are extracted and need to be converted into corresponding 3D point cloud data.

[0171] Step 5: Use the point cloud matching algorithm based on point cloud contour points to match the model point cloud with the scene point cloud to obtain the pose of the target workpiece.

[0172] This paper uses a point-pair matching algorithm based on point cloud contour points, leveraging the primary geometric features and relatively small number of contour point clouds to achieve higher computational efficiency and matching accuracy. Furthermore, curvature information is added to the existing point-pair features, making the features more complex and enabling better differentiation under similar working conditions for screen cabinet screws. The specific steps are:

[0173] (1) Point cloud contour point extraction, the steps are: first calculate the normal vector of each point through p i and its nearest neighbor set N i The mean of all points in Calculate p i The covariance matrix M i .

[0174]

[0175] Where n is p i The number of points in the field is M i Perform eigenvalue decomposition to obtain eigenvectors and eigenvalues.

[0176]

[0177] Where, is the eigenvector, is the characteristic value, let Then the minimum eigenvalue λ is the point p i The normal vector of each point is then found. A KD tree data structure can be used to find the neighboring points of each point. The angle between the normal vector of each point and the normal vectors of its neighboring points is then calculated. Points with angles exceeding a preset threshold are marked as contour points. Finally, the contour points can be optionally clustered to identify multiple different boundaries.

[0178] (2) Point-to-point feature matching based on point cloud contour points

[0179] This paper proposes a point-to-point feature matching algorithm (EPPF) based on point cloud contour points. The steps are:

[0180] a) Using a method based on normal vector angle, contour points are extracted from the point cloud model and scene point cloud respectively, and the contour points are used as key points for feature recognition, thereby reducing the number of key points and the calculation time, thereby improving the computational efficiency of the algorithm;

[0181] b) The point pair features are optimized, and curvature features are introduced to address the problem of point pairs with duplicate features. This point pair feature representation method only utilizes the point cloud normal and the line relationship between the point pairs as the feature representation, while also considering curvature information. This method can better distinguish different types of artifacts. In addition, the point pair features have good invariance and distinguishability, and remain consistent even in different poses and viewpoints. Therefore, even with large initial pose deviations, these feature descriptors can still provide reliable matching information, helping the algorithm find the correct correspondence.

[0182] c) Query the hash table to solve the local optimal solution problem caused by the KD tree.

[0183] d) Based on the query results of the hash table, complete the point cloud matching and obtain the target workpiece pose.

[0184] The point-to-point feature matching process based on point cloud contour points can be referenced Figure 2 .

[0185] The above point cloud matching results in a pose that is relatively close to the target. To obtain a more accurate pose, point cloud precise registration is required.

[0186] This paper proposes an improved closest point iterative algorithm for precise point cloud matching, incorporating information from previous iterations to accelerate the algorithm's convergence. Advantages include: predicting the next transformation using historical information, reducing oscillations and accelerating convergence; and automatically suppressing unstable iterations with weight coefficients, enhancing robustness.

[0187] The specific steps are:

[0188] Initialization: A coarse matching process is used to obtain the source point cloud P and the target point cloud Q. The initial transformation T0 = (R0, t0). Specifically, in this embodiment, the source point cloud refers to the point cloud of the scene to be recognized, i.e., the newly added point cloud; the target point cloud refers to the model point cloud. Therefore, the source and target point clouds here are the scene point cloud and the model point cloud, respectively, after coarse matching.

[0189] Set acceleration parameters, history window size m (take 4), convergence threshold ε, maximum number of iterations I max .

[0190] Step 1: Iterative optimization, repeat the following steps until convergence or the maximum number of iterations is reached;

[0191] For the current source point cloud P k Apply transformation T k , get the aligned point cloud P′ k ;T k Refers to the k-th transformation matrix, including rotation transformation and translation transformation.

[0192] Calculate the error function E(Δ) of the corresponding point pair and find the increment Δ that minimizes the error (k) ;

[0193] Maintain a historical transformation increment queue H = [Δ (0) ,Δ (1) ,…,Δ (k) ], length not exceeding m;

[0194] When the queue length exceeds m, remove the earliest increment;

[0195] Construct matrix A and vector b based on historical increments to solve the optimal weight β i :

[0196]

[0197] Solve β=(A T A) -1 A T b.

[0198] Calculate the weighted combination:

[0199]

[0200] Update the transformation based on the predicted increment:

[0201] T k+1 =T k +Δ pred (12)

[0202] Apply the new transformation T to the source point cloud P k+1 , and get P′ k+1 .

[0203] Step 2: Convergence judgment:

[0204] Calculate the current error (RMS error):

[0205]

[0206] If the RMSE change is less than the threshold ε or the maximum number of iterations is reached, the iteration is terminated.

[0207] Step 6: Use the hand-eye matrix to transform the position of the target screw into the robotic arm coordinate system, and control the robotic arm to operate the target screw.

[0208] The robot module integrates robot connection and gripping functions. Users can connect to the robot by setting its IP address. The system converts the poses obtained through point cloud matching into the robot's base coordinate system and uses signals to control the robot to locate screws on the display cabinet and perform maintenance tasks.

[0209] Embodiment 2: The present invention also provides a substation cabinet screw identification and positioning system based on three-dimensional vision, the system comprising:

[0210] An image acquisition module is used to obtain the conversion relationship between the camera coordinate system and the robotic arm coordinate system to complete hand-eye calibration. The camera captures screws in multiple positions to obtain corresponding images.

[0211] The effective area calculation module is used to normalize the brightness of the image to be processed, perform a differential operation between the background image and the current normalized image to be processed, and remove the invalid area in the differential image, so as to obtain the area of ​​interest, that is, the screen cabinet screw area, in the differential image;

[0212] The point cloud formation module is used to obtain the feature point matching relationship between the image to be processed and other images through image feature detection and matching methods, thereby obtaining the corresponding point cloud data;

[0213] A point cloud model construction module is used to obtain surface point cloud information of the screen cabinet screw based on the screen cabinet screw area, feature point matching relationship and point cloud data, screen the point cloud correspondence relationship through a random sampling consistency method, and obtain a point cloud pose transformation matrix based on the screened point cloud correspondence relationship, solve the optimal solution of the point cloud pose transformation matrix, and splice two frames of point clouds based on the obtained optimal solution; repeat this step to obtain a point cloud model of the screen cabinet screw;

[0214] The preprocessing module is used to remove irrelevant point clouds from the model point cloud using straight-through filtering and statistical filtering algorithms, downsample the point cloud data using an improved voxel filtering method, and then remove noise point clouds using an improved radius filtering method. Finally, the preprocessed point cloud of the cabinet screw end face is extracted.

[0215] The target screw pose calculation module is used to segment the two-dimensional target screw plane point cloud from the preprocessed point cloud, map its corresponding two-dimensional pixel coordinates to the three-dimensional point cloud data, extract contour points based on the three-dimensional point cloud data corresponding to the model point cloud, and propose contour points from the scene point cloud, thereby completing point cloud matching and obtaining the pose of the target screw;

[0216] The control module is used to convert the position of the target screw into the robotic arm coordinate system through the hand-eye matrix and control the robotic arm to operate the target screw.

[0217] Other technical features of the substation panel cabinet screw identification and positioning system based on three-dimensional vision described in this embodiment are similar to the corresponding substation panel cabinet screw identification and positioning method based on three-dimensional vision, and will not be repeated here.

[0218] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. "Multiple" means two or more, unless otherwise specifically defined.

[0219] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0220] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0221] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0222] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0223] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0224] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0225] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0226] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0227] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for identifying and locating screws in substation cabinets based on three-dimensional vision, characterized in that: The method includes: Obtaining the conversion relationship between the camera coordinate system and the robotic arm coordinate system to complete the hand-eye calibration, wherein the camera captures the screws in multiple positions to obtain corresponding images; Normalize the brightness of the image to be processed, perform a differential operation between the background image and the current normalized image to be processed, and remove the invalid area in the differential image to obtain the area of ​​interest, that is, the cabinet screw area; The feature point matching relationship between the image to be processed and other images is obtained through image feature detection and matching methods, thereby obtaining the corresponding point cloud data; The surface point cloud information of the screen cabinet screw is obtained based on the screen cabinet screw area, the feature point matching relationship and the point cloud data. The point cloud correspondence is screened by a random sampling consistency method. The point cloud pose transformation matrix is ​​obtained based on the screened point cloud correspondence. The optimal solution of the point cloud pose transformation matrix is ​​solved. The two frames of point clouds are spliced ​​based on the obtained optimal solution. This step is repeated to obtain a point cloud model of the screen cabinet screw. The straight-through filtering and statistical filtering algorithms are used to remove irrelevant point clouds from the model point cloud. The improved voxel filtering method is used to downsample the point cloud data. The improved radius filtering method is then applied to remove noise point clouds. Finally, the pre-processed point cloud of the cabinet screw end face is extracted. Segment the 2D model screw plane point cloud from the preprocessed point cloud, map it to 3D point cloud data according to its corresponding 2D pixel coordinates, extract contour points based on the 3D point cloud data corresponding to the model point cloud, and extract contour points from the scene point cloud to complete point cloud matching and obtain the position and pose of the target screw; The position of the target screw is converted to the robotic arm coordinate system through the hand-eye matrix, and the robotic arm is controlled to operate the target screw.

2. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 1 is characterized in that: The brightness normalization of the image to be processed includes: Calculate the average grayscale value of each frame in the first N frames, and record the image corresponding to the median of the average grayscale value as M; Calculate the average pixel value μ of image M m and pixel value standard deviation σ m And the average pixel value μ of the image P to be processed p and pixel value standard deviation σ p ; Thus, the brightness normalization coefficient α is obtained, and the calculation formula is as follows: min((am m -m p ) 2 +(a 2 s m -s p ) 2 ),α∈[0,1]; Where, (αμ m -μ p ) 2 Represents the difference between the average pixel values ​​of M and P, (α 2 σ m -σ p ) 2 Represents the difference between the standard deviations of the pixel values ​​of M and P; Multiply all pixel values ​​of image P by the brightness normalization coefficient α so that the average brightness of M and P are close and the brightness distribution is similar, thereby completing image normalization.

3. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 2 is characterized in that: The step of performing a difference operation on the background image and the currently normalized image to be processed and removing invalid areas in the difference image includes: Traverse the differential image, and for the current pixel point, if it has been marked as belonging to a connected area, skip it; otherwise, use the current point as the seed to start growing the connected area. Through the breadth-first search algorithm, all other pixel points connected to the seed point are found, marked with a new area number, and the number of pixels in the current area is counted. The connected area formed is traversed, and the point with the largest number of pixel values ​​is taken as the area of ​​interest, that is, the screen cabinet screw area.

4. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 1 is characterized in that: The method of obtaining a feature point matching relationship between the image to be processed and other images by an image feature detection and matching method, thereby obtaining corresponding point cloud data, includes: Since each point in the point cloud is generated by a pixel in the grayscale image, each point in the point cloud can find a corresponding pixel in the grayscale image. Based on this one-to-one correspondence, the matching relationship between the pixels in the two grayscale images is obtained and then mapped to the point cloud, and the corresponding relationship between the points in the two point clouds is obtained, thereby obtaining the corresponding point cloud data between the point clouds.

5. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 4 is characterized in that: Obtaining a point cloud pose transformation matrix based on the filtered point cloud correspondence relationship and solving the optimal solution of the point cloud pose transformation matrix includes: The objective function is solved by a nonlinear optimization algorithm based on Levenberg-Marquardt, thereby obtaining the optimal solution of the point cloud pose transformation matrix.

6. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 1 is characterized in that: The method of removing irrelevant point clouds from the model point cloud by using the straight-through filtering and statistical filtering algorithms includes: In the cabinet screw work scenario, a bounding box is constructed and the maximum and minimum values ​​of the x, y, and z coordinates are set. Based on the set threshold range, data within the specified range is filtered and retained, and data outside the range is treated as outliers and removed. Statistical filtering algorithms are used to reduce high-frequency noise in point cloud data.

7. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 6 is characterized in that: The method of reducing high-frequency noise in point cloud data by using a statistical filtering algorithm includes: For each point p in the model point cloud i , determine its neighborhood N, the neighborhood N is based on point p i A sphere of fixed size centered at i The average distance d of points in the neighborhood mean and standard deviation d std , respectively: Judgment point p i Is it an outlier: For each point, if the distance d from the point in the neighborhood to it is p If the difference between the average distance and the point exceeds a certain threshold, it will be marked as an outlier. The judgment condition is: |d p -d mean |>k*d std , k is the definition threshold; Remove outliers: Remove or replace points marked as outliers from the point cloud to obtain a new point cloud that does not contain outliers.

8. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 6 is characterized in that: The improved voxel filtering method is used to complete the downsampling processing of the point cloud data, including: Set the voxel size corresponding to the new point cloud data to N; divide the point cloud data into sub-regions with voxels as the unit, and each voxel is regarded as a cube or cube grid unit; For each voxel, calculate the centroid of all points in it, that is, the average value of the coordinates, and let the point set in the voxel be P v ={p1,p2,…,p i }, where p i is the coordinate of the i-th point, then the centroid p c The calculation formula is: The entire point cloud constructs a KD tree to quickly find the point closest to the centroid; for each voxel, the distance to the centroid p is found through the KD tree. c The nearest point p cr ; Use the nearest point p cr Replace all point clouds in a voxel, that is, all points p in a voxel i Replace with the nearest point p cr ; Repeat the above steps for all voxels until all voxels have been processed.

9. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 8 is characterized in that: The improved radius filtering method is applied to remove noise point clouds and finally extract the pre-processed point clouds of the screen cabinet screw end faces, including: Assume that the input point cloud data set is Q, the filter radius is d, the quantity threshold is k, and Q is expressed as follows: Q={q1,q2,…,q n }; Select any point q in Q i As the search seed point, with d as the search radius, count the number of points in the neighborhood m, if the relationship between m and k meets the requirements, then keep it as q i ', m represents the following: In the formula, B(q i ,r) indicates q i is the center of the sphere and r is the number of data points in the spherical space with radius; Repeat the above steps to traverse each point in the set Q, and reorganize all points that meet the requirements into the point cloud set Q i '; Q i '={q'1,q'2,…,q j }={q' i ∈Q|B(q' i ,r)>k}。 10. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 1, characterized in that: The extracting of contour points according to the three-dimensional point cloud data corresponding to the model point cloud comprises: First, calculate the normal vector of each point in the model point cloud, through p i and its nearest neighbor set N i The mean of all points in Calculate p i The covariance matrix M i , expressed as: Where n is p i The number of points in the field is M i Perform eigenvalue decomposition to obtain eigenvectors and eigenvalues; Where, is the eigenvector, is the characteristic value, let Then the minimum eigenvalue λ is the point p i The normal vector of Then find the adjacent points of each point, and then calculate the angle between the normal vector of each point and the normal vector of its adjacent points. Points whose angle exceeds a preset threshold are marked as contour points.

11. The method for identifying and locating screws in a substation cabinet based on three-dimensional vision according to claim 10, characterized in that: The method of extracting contour points from the scene point cloud to complete point cloud matching and obtain the position and posture of the target screw includes: rough matching and fine matching of point-to-point features of contour points. The fine matching of point-to-point features of contour points includes: Initialization: Obtain source point cloud P and target point cloud Q through coarse matching, initial transformation T0 = (R0, t0), set acceleration parameters, history window size m, convergence threshold ε, maximum number of iterations I max ; Iterative optimization repeats the following steps until convergence or the maximum number of iterations is reached: For the current source point cloud P k Apply transformation T k , get the aligned point cloud P' k ; Calculate the error function E(Δ) of the corresponding point pair and solve the increment Δ that minimizes the error (k) ; Maintain a historical transformation increment queue H = [Δ (0) ,Δ (1) ,…,Δ (k) ], the length does not exceed m; when the queue length exceeds m, remove the earliest increment; Construct matrix A and vector b based on historical increments to solve the optimal weight β i : Solve β=(A T A) -1 A T b; Calculate the weighted combination: Update the transformation based on the predicted increment: T k+1 =T k +Δ pred ; Apply the new transformation T to the source point cloud P k+1 , and get P' k+1 ; Convergence judgment: Calculate the current error RMSE. If the RMSE change is less than the threshold ε or the maximum number of iterations is reached, the iteration is terminated.

12. A three-dimensional vision-based substation cabinet screw identification and positioning system, characterized in that: The system includes: An image acquisition module is used to obtain the conversion relationship between the camera coordinate system and the robotic arm coordinate system to complete hand-eye calibration. The camera captures screws in multiple positions to obtain corresponding images. The effective area calculation module is used to normalize the brightness of the image to be processed, perform a differential operation between the background image and the current normalized image to be processed, and remove the invalid area in the differential image, so as to obtain the area of ​​interest, that is, the screen cabinet screw area, in the differential image; The point cloud formation module is used to obtain the feature point matching relationship between the image to be processed and other images through image feature detection and matching methods, thereby obtaining the corresponding point cloud data; A point cloud model construction module is used to obtain surface point cloud information of the screen cabinet screw based on the screen cabinet screw area, feature point matching relationship and point cloud data, screen the point cloud correspondence relationship through a random sampling consistency method, and obtain a point cloud pose transformation matrix based on the screened point cloud correspondence relationship, solve the optimal solution of the point cloud pose transformation matrix, and splice two frames of point clouds based on the obtained optimal solution; repeat this step to obtain a point cloud model of the screen cabinet screw; The preprocessing module is used to remove irrelevant point clouds from the model point cloud using straight-through filtering and statistical filtering algorithms, downsample the point cloud data using an improved voxel filtering method, and then remove noise point clouds using an improved radius filtering method. Finally, the preprocessed point cloud of the cabinet screw end face is extracted. The target screw pose calculation module is used to segment the two-dimensional target screw plane point cloud from the preprocessed point cloud, map its corresponding two-dimensional pixel coordinates to the three-dimensional point cloud data, extract contour points based on the three-dimensional point cloud data corresponding to the model point cloud, and propose contour points from the scene point cloud, thereby completing point cloud matching and obtaining the pose of the target screw; The control module is used to convert the position of the target screw into the robotic arm coordinate system through the hand-eye matrix and control the robotic arm to operate the target screw.

Images