Large-scale workpiece size detection system based on multi-sensor data fusion

Abstract

The invention relates to the technical field of point cloud circle center identification, and provides a multi-sensor data fusion large workpiece size detection system, the large workpiece size detection system comprises a three-dimensional camera, a PLC, a position sensor, a mechanical arm, a point cloud splicing module and a size detection module, the three-dimensional camera is installed at the tail end of the mechanical arm; the position sensor is used for detecting the position of a workpiece to be detected and sending a trigger signal to the PLC; after receiving the trigger signal, the PLC sends a control instruction to the mechanical arm; the mechanical arm is used for acquiring corresponding first point cloud data and second point cloud data from the upper part and the lower part of each workpiece part according to the control instruction; the point cloud splicing module is used for acquiring the first point cloud data and the second point cloud data, and splicing and fusing the first point cloud data and the second point cloud data to obtain third point cloud data; the size detection module is used for calculating the size of the to-be-detected workpiece. According to the invention, a plurality of sensors work cooperatively, and measurement data at different angles are collected, processed and fused, so that a large-scale workpiece dimension detection result is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud center recognition, and in particular to a large workpiece size detection system for multi-sensor data fusion. Background Art

[0002] Currently, large workpiece size detection systems usually use a single sensor for detection, such as a laser rangefinder, a single three-dimensional camera, etc. Although these sensors can achieve the size detection function to a certain extent when used alone, there are obvious limitations in actual applications.

[0003] The accuracy of a single sensor is limited by its working principle and technical characteristics. For example, although a laser rangefinder has high accuracy, when measuring a large and complex workpiece, due to the complexity of the workpiece surface shape, measurement errors may accumulate, making it difficult to meet the high-precision detection requirements; when measuring some key parts of a large workpiece, a traditional single-sensor system may not be able to provide sufficient detailed information, thus affecting the overall measurement accuracy.

[0004] A single sensor is easily affected by changes in environmental conditions, such as temperature, humidity, light, etc. These environmental factors may cause the performance of the sensor to decline, thus affecting the stability of the measurement results; the error accumulation of the sensor itself is also a problem. During a long-term measurement process, the errors of a single sensor may gradually accumulate, resulting in inaccurate measurement results.

[0005] Some single sensors may not be able to meet the requirements of large workpiece size detection due to the limitation of their measurement range. For example, the measurement range of some laser sensors is limited, and some parts of a large workpiece may not be effectively measured; in order to cover all the detection areas of a large workpiece, multiple sensors may need to be used in combination, but this combination method often lacks an effective data fusion mechanism, resulting in inconsistent measurement results. Summary of the Invention

[0006] Aiming at the above defects, the purpose of the present invention is to propose a large workpiece size detection system for multi-sensor data fusion, aiming to have multiple sensors work together, and generate high-precision three-dimensional point clouds and size detection results by collecting, processing, and fusing measurement data from different angles.

[0007] To achieve this purpose, the present invention adopts the following technical solutions:

[0008] A large workpiece size detection system for multi-sensor data fusion, the large workpiece size detection system includes a three-dimensional camera, a PLC, a position sensor, a robotic arm, a point cloud stitching module, and a size detection module, and the three-dimensional camera is installed at the end of the robotic arm;

[0009] The position sensor is used for the position of the workpiece to be detected, and when the workpiece to be detected enters the detectable area, it sends a trigger signal to the PLC according to the position of the workpiece to be detected;

[0010] After receiving the trigger signal, the PLC sends a control instruction to the robotic arm;

[0011] The robotic arm is used to receive and parse the control instruction, move according to the preset motion trajectory and motion posture. When the 3D camera moves to the part to be detected of the workpiece to be detected, the 3D camera obtains corresponding first point cloud data and second point cloud data from above and below each workpiece part respectively;

[0012] The point cloud stitching module is used to obtain the first and second point cloud data, roughly align the feature points in the first and second point cloud data according to the pose information of the 3D camera when obtaining the point cloud data, and based on the KD-Tree constructed from the second point cloud data, finely align the feature points of the roughly aligned first and second point cloud data, and splice and fuse the first and second point cloud data to obtain third point cloud data;

[0013] The dimension detection module is used to select key points, straight line measurement areas and plane fitting areas in the third point cloud data, calculate the distances between the key points, fit a straight line in the straight line measurement area, fit a plane in the plane fitting area, and calculate the distance and angle between planes, and convert the third point cloud data and the data calculated by the dimension detection module to the same coordinate system to obtain the dimensions of the workpiece to be detected.

[0014] Preferably, the pose information of the 3D camera includes a rotation matrix and a relative displacement amount;

[0015] Roughly aligning the feature points in the first and second point cloud data according to the pose information of the 3D camera when obtaining the point cloud data includes:

[0016] Obtaining the relative rotation matrix and relative displacement vector of the second point cloud data relative to the first point cloud data according to the rotation matrix and relative displacement amount of the 3D camera, satisfying the relationship:

[0017]

[0018] Transform each point in the second point cloud data, and roughly align the point cloud data, satisfying the relationship:

[0019] P B ′ =R BA P B +t BA ;

[0020] where, R BArepresents the relative rotation matrix of the second point cloud data with respect to the first point cloud data, t BA represents the relative displacement vector of the second point cloud data with respect to the first point cloud data, represents the transpose of the rotation matrix when the 3D camera acquires the first point cloud data, R B represents the rotation matrix when the 3D camera acquires the second point cloud data, t B represents the displacement amount when the 3D camera acquires the second point cloud data, t A represents the displacement amount when the 3D camera acquires the first point cloud data, P B ′ represents the second point cloud data after rough alignment, P B represents the second point cloud data before rough alignment.

[0021] Further, the fine alignment of the feature points of the first and second point cloud data after rough alignment according to the KD-Tree constructed based on the second point cloud data includes:

[0022] Select an axis as the splitting axis;

[0023] Sort the points in the second point cloud data according to the coordinate values of the splitting axis, select the median point as the splitting point, and divide the second point cloud data into left and right subtrees, where the left subtree contains all the points with coordinate values less than the splitting point, and the right subtree contains all the points with coordinate values greater than or equal to the splitting point;

[0024] Recursively repeat the above steps for the left and right subtrees until the number of points in the subtrees is less than a preset threshold, to obtain the KD-Tree constructed based on the second point cloud data;

[0025] Starting from the root node of the KD-Tree, by comparing the coordinate values of the points in the first point cloud data on the splitting axis with the splitting point coordinate values of the current node, enter the left or right subtree to continue the search. When reaching a leaf node, take the point corresponding to the leaf node as the current nearest neighbor point candidate;

[0026] Backtrack to the parent node, check whether there is a closer point in the other subtree of the current node. If the other subtree of the current node intersects with the hypersphere centered at the point with the distance of the current nearest neighbor point candidate as the radius, then it is necessary to continue the search in the current subtree and update the nearest neighbor point candidate;

[0027] Repeat the above backtracking and checking steps until the entire KD-Tree is traversed, to obtain the nearest neighbor points of the first point cloud data in the second point cloud data.

[0028] Preferably, the pose information of the 3D camera includes the rotation matrix and the relative displacement amount;

[0029] The steps for obtaining the pose information of the 3D camera when acquiring point cloud data include:

[0030] Obtain the centroids of the first point cloud data and the second point cloud data, and subtract their centroids from the first point cloud data and the second point cloud data respectively to obtain the centroid-removed points;

[0031] Calculate the rotation matrix R and H based on the centroid-removed points of the first point cloud data and the second point cloud data, satisfying the relational expression:

[0032]

[0033] Perform singular value decomposition on H to obtain H = U∑V T , and then obtain the rotation matrix R according to R = VU T Obtain the rotation matrix R, and calculate the relative displacement t according to the rotation matrix R and the centroids of the first and second point cloud data, satisfying the relational expression:

[0034] The obtained rotation matrix R and relative displacement t satisfy the following relational expression:

[0035]

[0036] where E(R, t) represents the objective function, p Ai represents the i-th point of the first point cloud data, p Bi represents the i-th point of the second point cloud data, q Ai represents the i-th centroid-removed point of the first point cloud data, q Bi represents the i-th centroid-removed point of the second point cloud data, and respectively represent the centroids of the first and second point cloud data, and U and V represent orthogonal matrices.

[0037] Preferably, the step of splicing and fusing the first and second point cloud data to obtain the third point cloud data includes:

[0038] Set a distance threshold, and filter out the overlapping point pairs with the point cloud spacing within the distance threshold in the first and second point cloud data according to the search structure;

[0039] Calculate the distances from each point in the overlapping point pairs to the reference center or the splicing boundary, and use the distances as the weight basis to obtain the weights of each point in the overlapping point pairs;

[0040] Calculate the weighted average coordinates of the overlapping point pairs according to the weights of each point in the overlapping point pairs, satisfying the relational expression:

[0041]

[0042] where W A and WB respectively represent the weights of the points in the first point cloud data and the second point cloud data in the overlapping point pair, P A and P B respectively represent the coordinates of the points in the first point cloud data and the second point cloud data in the overlapping point pair, d A and d B respectively represent the distances from the points in the first point cloud data and the second point cloud data in the overlapping point pair to the reference center or the splicing boundary;

[0043] Retain the non-overlapping points in the first point cloud data and the second point cloud data, and replace the overlapping point pairs with weighted average coordinate points to obtain the third point cloud data.

[0044] Preferably, the selection of key points in the third point cloud data includes:

[0045] Traverse and calculate the gradients of each point in the third point cloud data in the X, Y, and Z directions, and calculate the structure tensor of each point based on the gradients of each point in the X, Y, and Z directions;

[0046] Calculate the Harris response value according to the structure tensor of each point, set the Harris response threshold. If the Harris response value of a certain point is greater than the Harris response threshold and its Harris response value is the maximum value in the field, then this point is a corner point;

[0047] Analyze the eigenvalues of the structure tensor of each point. If there is a certain point whose first eigenvalue is much larger than the second eigenvalue and the third eigenvalue, and the difference between the second eigenvalue and the third eigenvalue is less than the preset difference threshold, then regard this point as an edge point;

[0048] If the workpiece to be detected is symmetric in shape, then calculate the centroid in the third point cloud data to determine the center point.

[0049] Preferably, the fitting of a straight line in the straight line measurement area includes:

[0050] Set the parametric equation of the straight line to be fitted where represents a point on the straight line, represents the direction vector of the straight line;

[0051] Based on minimizing the sum of the squares of the distances from the points to the straight line in the parametric equation, the parameters of the straight line can be obtained, satisfying the relationship:

[0052]

[0053] where, represents a certain point in the straight line measurement area, represents a point on the straight line, represents the direction vector of the straight line.

[0054] Preferably, converting the third point cloud data and the data calculated by the dimension detection module into the same coordinate system includes:

[0055] Setting a homogeneous transformation matrix for describing the transformation from the base coordinate system of the robotic arm to the end effector coordinate system of the robotic arm

[0056] Converting the third point cloud data from the camera coordinate system to the end effector coordinate system of the robotic arm, satisfying the relationship:

[0057]

[0058] where P arm is the representation of the third point cloud data in the end effector coordinate system of the robotic arm, represents the transformation matrix from the camera coordinate system to the end effector coordinate system of the robotic arm, and P cam is the representation of the third point cloud data in the camera coordinate system arm;

[0059] Converting the third point cloud data from the end effector coordinate system of the robotic arm to the base coordinate system of the robotic arm, satisfying the relationship:

[0060] One of the above technical solutions has the following advantages or beneficial effects:

[0061] Through the collaborative work of the structured light 3D camera, position sensor, robotic arm, PLC, point cloud stitching module and dimension detection module, the present invention realizes high-precision and automated dimension detection of large workpieces. The system can use two structured light 3D cameras to obtain the 3D point cloud data of the workpiece from different angles, and precisely fuse the multi-view data through the point cloud stitching algorithm to generate a complete workpiece model; the combination of the position sensor and the PLC realizes the automatic start of the detection process, and the flexible movement ability of the robotic arm ensures the efficiency of collecting data from multiple key parts; the dimension detection module accurately measures the dimension features of the workpiece through key point extraction, line and plane (or surface) fitting algorithms, and converts all data into a unified coordinate system to output high-precision detection results; the present invention can significantly improve the detection accuracy and coverage, overcome the limitations of traditional single-sensor systems through multi-sensor fusion; the efficient point cloud stitching algorithm improves the efficiency and accuracy of data processing; and the high-precision dimension detection algorithm based on 3D data meets the measurement requirements of complex geometric shapes; in addition, it also has better environmental adaptability and stability, can maintain high-precision measurement in complex industrial scenarios, can significantly improve the detection efficiency and production quality, and is applicable to multiple fields such as automobile manufacturing, aerospace, and large medical equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0063] Figure 1 is a schematic structural diagram of a large workpiece size detection system for multi-sensor data fusion provided by an embodiment of the present invention;

[0064] Figure 2 is a flowchart of the operation of a large workpiece size detection system for multi-sensor data fusion provided by another embodiment of the present invention. Detailed implementation manners

[0065] The following will describe in detail the embodiments of the present invention. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0066] In the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. Without further limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0067] A large workpiece size detection system for multi-sensor data fusion, as Figure 1 shown, a preferred embodiment of the present invention, the large workpiece size detection system includes a 3D camera, a PLC, a position sensor, a robotic arm, a point cloud stitching module and a size detection module. The 3D camera is installed at the end of the robotic arm;

[0068] The position sensor is used to detect the position of the workpiece to be detected, and when the workpiece to be detected enters the detectable area, it sends a trigger signal to the PLC according to the position of the workpiece to be detected;

[0069] After receiving the trigger signal, the PLC sends a control instruction to the robotic arm;

[0070] The robotic arm is used to receive and parse the control instruction, move according to a preset motion trajectory and motion posture, and when the 3D camera moves to the part to be detected of the workpiece to be detected, the 3D camera obtains corresponding first point cloud data and second point cloud data from above and below each workpiece part respectively;

[0071] The point cloud stitching module is used to obtain the first and second point cloud data, roughly align the feature points in the first and second point cloud data according to the pose information of the 3D camera when obtaining the point cloud data, and based on the KD-Tree constructed from the second point cloud data, finely align the feature points of the roughly aligned first and second point cloud data, and splice and fuse the first and second point cloud data to obtain third point cloud data;

[0072] The dimension detection module is used to select key points, straight line measurement areas and plane fitting areas in the third point cloud data, calculate the distances between the key points, fit a straight line in the straight line measurement area, fit a plane in the plane fitting area, and calculate the distance and angle between planes, and convert the third point cloud data and the data calculated by the dimension detection module to the same coordinate system to obtain the dimensions of the workpiece to be detected.

[0073] Specifically, the position sensor is a component in the system used to detect the position of the workpiece. It detects whether the workpiece enters the detectable area through the principle of light occlusion and sends a trigger signal to the PLC to start the detection process. The role of the position sensor is to ensure that the detection process is started in a timely manner when the workpiece reaches the specified position, avoid ineffective detection, and improve the detection efficiency and reliability of the system.

[0074] The position sensor triggers the detection process by detecting whether the workpiece enters the detectable area. It can utilize the principle of light occlusion. When the workpiece moves into the detection area of the sensor, the workpiece blocks the light, resulting in a change in the amount of light received by the sensor, thereby generating a trigger signal. The sensitivity and response speed of the position sensor directly affect the start accuracy and timeliness of the detection process and are one of the key components for the system to achieve automated detection.

[0075] The implementation method of the position sensor can select different types according to the application scenario and accuracy requirements. For example, an optoelectronic sensor can be used to detect the workpiece position through the principle of light occlusion, which is suitable for most industrial scenarios. For higher accuracy requirements, a lidar sensor can be used to detect the workpiece position through the principle of distance measurement, providing higher accuracy and stronger environmental adaptability. In addition, multiple sensor technologies, such as ultrasonic sensors or magnetic sensors, can be combined to meet different detection needs.

[0076] The PLC is the core control unit in the system for coordinating the actions of various components. It receives signals from the position sensors and sends control instructions to the robotic arm to ensure the automation and accuracy of the detection process. The role of the PLC is to serve as the "brain" of the system, controlling devices such as the robotic arm and sensors through programming to ensure the smooth progress of the entire detection process. The functions performed by the PLC can be achieved by the position sensors in actual applications, such as Figure 2 as shown, directly sending instructions to the robotic arm by the position sensors.

[0077] The robotic arm is the component in the system for driving the 3D camera to move. According to the control instructions of the PLC, the robotic arm moves along the preset trajectory and posture to ensure that the 3D camera can reach the part of the workpiece to be measured. The role of the robotic arm is to provide flexible movement capabilities, enabling the system to perform multi-angle inspections on complex workpieces and improving the inspection coverage and accuracy. The performance of the robotic arm directly affects the inspection efficiency and data quality of the system. In this embodiment, there are two robotic arms, and different types of robotic arms can be selected according to the application scenario and accuracy requirements. For example, industrial-grade six-axis robotic arms can be used to achieve high-precision inspection tasks through precise motion control. For scenarios that require human-robot collaboration, collaborative robots can be used to achieve efficient inspections through flexible movement capabilities and safety features. In addition, the movement trajectory and posture of the robotic arm can be optimized according to the shape of the workpiece and inspection requirements to improve the inspection efficiency and data quality.

[0078] The 3D camera is the key device in the system for obtaining the three-dimensional data of the workpiece. It projects and captures the light stripe pattern to measure the shape and size of the workpiece surface. In this embodiment, there are two 3D cameras, which are respectively installed at the ends of their respective independent robotic arms. One robotic arm is responsible for moving the carried 3D camera above the large workpiece, and the other robotic arm moves the corresponding 3D camera below the workpiece. The 3D camera provides basic data for subsequent point cloud stitching and dimensional inspection, and its accuracy directly affects the inspection accuracy of the entire system. The performance of the 3D camera determines the quality of the point cloud data that the system can obtain, thereby affecting the accuracy and reliability of the final dimensional inspection.

[0079] The 3D camera projects a specific light stripe pattern onto the workpiece surface through structured light technology. When the pattern is deformed due to the shape of the workpiece, the camera captures these deformed patterns. Using 3D imaging algorithms and known camera parameters, the three-dimensional coordinate information of each point on the workpiece surface is calculated to generate high-precision point cloud data. This process not only depends on the optical system of the camera but also requires precise algorithms to process the conversion between pattern deformation and three-dimensional coordinates to ensure that the point cloud data obtained from different angles can accurately reflect the geometric features of the workpiece.

[0080] The point cloud stitching module acquires the point cloud data collected by the 3D camera and uses the pose information of the camera to perform rough alignment and fine alignment on the point cloud data. First, key feature points (such as edges or corners) are found in the point cloud through a feature point extraction algorithm, and these feature points are used for initial alignment. Then, through the construction of a KD-Tree and the nearest neighbor search algorithm, the coarsely aligned point cloud is finely aligned, and finally a complete point cloud model is generated. Among them, the traditional ICP (Iterative Closest Point) algorithm can be used for point cloud stitching, and high-precision alignment is achieved by minimizing the distance error between point clouds. For complex point cloud data, deep learning algorithms can be combined to optimize the stitching process, improve the robustness and efficiency of the algorithm. In addition, the stitching accuracy and speed can be further improved by optimizing the feature point extraction algorithm and the nearest neighbor search algorithm.

[0081] The dimension detection module analyzes the complete point cloud data generated by the point cloud stitching module, extracts key points, straight line measurement areas, and plane fitting areas, and uses algorithms such as distance calculation between key points, straight line fitting, and plane fitting to accurately measure the dimension features of the workpiece, such as length, width, thickness, flatness, and angle. By converting all measurement data to a unified coordinate system, the dimension detection module can output high-precision dimension detection results, which not only rely on accurate point cloud data but also require efficient algorithms to process complex geometries to ensure the accuracy and reliability of the measurement results.

[0082] The dimension detection module can select different types according to the application scenario and accuracy requirements. For workpieces with regular geometries, the classical least squares method can be used for straight line and plane fitting, and dimension measurement is achieved by calculating the Euclidean distance between key points. For workpieces with complex geometries, deep learning algorithms can be combined, and features are automatically extracted and dimensions are calculated through a convolutional neural network (CNN) to improve the measurement accuracy and robustness. The detection efficiency and adaptability can also be further improved by optimizing the algorithm and data processing flow. For example, multi-scale analysis methods are used to process point cloud data with different resolutions, or prior knowledge (such as the CAD model of the workpiece) is introduced to assist detection to further improve the measurement accuracy.

[0083] Preferably, the pose information of the 3D camera includes a rotation matrix and a relative displacement amount;

[0084] Coarsely aligning the feature points in the first and second point cloud data according to the pose information of the 3D camera when acquiring the point cloud data includes:

[0085] Obtaining the relative rotation matrix and relative displacement vector of the second point cloud data relative to the first point cloud data according to the rotation matrix and relative displacement amount of the 3D camera, satisfying the relationship:

[0086]

[0087] Transform each point in the second point cloud data to perform a rough alignment between the point cloud data, satisfying the relationship:

[0088] P B ′ = R BA P B + t BA ;

[0089] wherein, R BA represents the relative rotation matrix of the second point cloud data with respect to the first point cloud data, and t BA represents the relative displacement vector of the second point cloud data with respect to the first point cloud data, represents the transpose of the rotation matrix when the 3D camera acquires the first point cloud data, R B represents the rotation matrix when the 3D camera acquires the second point cloud data, t B represents the displacement amount when the 3D camera acquires the second point cloud data, t A represents the displacement amount when the 3D camera acquires the first point cloud data, P B ′ represents the second point cloud data after rough alignment, and P B represents the second point cloud data before rough alignment.

[0090] Specifically, when performing rough alignment, based on the pose information of the 3D camera (including the rotation matrix and displacement amount), the point cloud data obtained from two different perspectives are roughly aligned. By calculating the relative rotation matrix R BA and the relative displacement vector t BA of the second point cloud data with respect to the first point cloud data, and using these transformation parameters to transform each point in the second point cloud data, so that the two point cloud data can be roughly aligned to the same coordinate system, that is, using the nearest neighbor search algorithm (such as KD-Tree), match between the feature descriptors of the first point cloud data and the second point cloud data. For each feature point P Ai in the first point cloud data, find the feature point P Bi in the second point cloud data whose feature descriptor is the most similar to it, and form a matching point pair (P Ai , P Bi ).

[0091] Therefore, using the pose information of the 3D camera to perform rough alignment on the point cloud data provides a good initial state for subsequent fine alignment, can reduce the number of iterations of the ICP algorithm, and improve the efficiency and accuracy of point cloud stitching.

[0092] Furthermore, the fine alignment of the feature points of the first and second point cloud data after rough alignment according to the KD-Tree constructed based on the second point cloud data includes:

[0093] Select an axis as the splitting axis;

[0094] Sort the points in the second point cloud data according to the coordinate values of the splitting axis, select the median point as the splitting point, and divide the second point cloud data into two subtrees, the left subtree containing all points with coordinate values less than the splitting point, and the right subtree containing all points with coordinate values greater than or equal to the splitting point;

[0095] Recursively repeat the above steps for the left and right subtrees until the number of points in the subtrees is less than a preset threshold, obtaining a KD-Tree constructed based on the second point cloud data;

[0096] Starting from the root node of the KD-Tree, by comparing the coordinate values of the points in the first point cloud data on the splitting axis with the splitting point coordinate values of the current node, enter the left or right subtree to continue the search. When reaching a leaf node, take the point corresponding to the leaf node as the current nearest neighbor candidate;

[0097] Backtrack to the parent node and check whether there are closer points in the other subtree of the current node. If the other subtree of the current node intersects with the hypersphere centered at the point with the distance of the current nearest neighbor candidate as the radius, then it is necessary to continue the search in the current subtree and update the nearest neighbor candidate;

[0098] Repeat the above backtracking and checking steps until the entire KD-Tree is traversed to obtain the nearest neighbor points of the first point cloud data in the second point cloud data.

[0099] Specifically, using the KD-Tree (K-dimensional tree) data structure constructed based on the second point cloud data, perform fine alignment on the feature points in the coarsely aligned first point cloud data and the second point cloud data. The KD-Tree recursively divides the point cloud data into left and right subtrees, forming an efficient spatial partitioning structure, thereby accelerating the nearest neighbor search process. By starting from the root node of the KD-Tree and comparing the points in the first point cloud data with the splitting point coordinate values of the current node layer by layer, enter the corresponding subtree to continue the search, and finally find the points in the second point cloud data that are closest to each point in the first point cloud data, which can significantly improve the alignment accuracy of the point cloud data and provide high-precision matching point pairs for subsequent point cloud stitching.

[0100] In this step, the KD-Tree is an efficient spatial search data structure used to accelerate the nearest neighbor search in a multi-dimensional space. It recursively divides the point cloud data into left and right subtrees by selecting an axis as the splitting axis. The left subtree contains all points with coordinate values less than the splitting point, and the right subtree contains all points with coordinate values greater than or equal to the splitting point. Through this recursive division, the KD-Tree can quickly locate the nearest neighbor of the target point. During the fine alignment process, each point in the first point cloud data finds its nearest neighbor in the second point cloud data through the KD-Tree search, thus achieving high-precision alignment of the two point cloud data. Therefore, it can not only improve the search efficiency but also ensure the accuracy and reliability of the alignment.

[0101] Preferably, the pose information of the three-dimensional camera includes a rotation matrix and a relative displacement amount;

[0102] The steps for obtaining the pose information of the three-dimensional camera when acquiring the point cloud data include:

[0103] Obtain the centroids of the first point cloud data and the second point cloud data, and subtract their centroids from the first point cloud data and the second point cloud data respectively to obtain the centroid-removed points;

[0104] Calculate the rotation matrix R and H based on the centroid-removed points of the first point cloud data and the second point cloud data, satisfying the relational expression:

[0105]

[0106] Perform singular value decomposition on H to obtain H = U∑V T , and then obtain the rotation matrix R according to R = VU T Obtain the rotation matrix R, and calculate the relative displacement amount t based on the rotation matrix R and the centroids of the first and second point cloud data, satisfying the relational expression:

[0107] The obtained rotation matrix R and relative displacement amount t satisfy the following relational expression:

[0108]

[0109] where E(R,t) represents the objective function, p Ai represents the i-th point of the first point cloud data, p Bi represents the i-th point of the second point cloud data, q Ai represents the i-th centroid-removed point of the first point cloud data, q Bi represents the i-th centroid-removed point of the second point cloud data, and respectively represent the centroids of the first and second point cloud data, and U and V represent orthogonal matrices.

[0110] In this embodiment, the rotation matrix R and the relative displacement t between two point cloud data are calculated through centroid processing and singular value decomposition (SVD) of the point cloud data. First, the centroids of the first point cloud data and the second point cloud data are calculated, and the centroid removal processing is performed on the point cloud data to eliminate the influence of translation. Then, the matrix H is calculated using the centroid-removed points, and the orthogonal matrices U and V are obtained through singular value decomposition, thereby calculating the rotation matrix R = VU T Finally, the relative displacement is calculated based on the rotation matrix and the centroid information The goal of this process is to find the rotation matrix R and the relative displacement t that can minimize the distance error E(R, t) between the point cloud data, efficiently calculate the rotation and translation relationships between the two point cloud data, and provide accurate transformation parameters for the alignment of the point cloud data. By optimizing the algorithm of singular value decomposition and improving the calculation efficiency, the performance of this step can be further improved to make it applicable to the processing of large-scale point cloud data

[0111] In this step, the centroid refers to the geometric center of the point cloud data. By subtracting the centroid from the point cloud data, the influence of translation can be eliminated, making the point cloud data centered at the origin. Centroid removal means subtracting the centroid from each point in the point cloud data to obtain the point cloud data centered at the origin. The matrix H is calculated from the centroid-removed point cloud data and is used to describe the correlation between the two point cloud data. Singular value decomposition (SVD) is a matrix decomposition method that decomposes the matrix H into U∑V T where U and V are orthogonal matrices used to calculate the rotation matrix R. The rotation matrix R describes the rotation relationship between the two point cloud data, and the relative displacement t describes the translation relationship between the two point cloud data. These parameters are jointly used to align the two point cloud data into the same coordinate system, providing an accurate transformation relationship for subsequent point cloud stitching

[0112] Preferably, the splicing and fusion of the first and second point cloud data to obtain the third point cloud data includes:

[0113] Set a distance threshold, and filter out the overlapping point pairs with the point cloud spacing within the distance threshold in the first and second point cloud data according to the search structure

[0114] Calculate the distances from each point in the overlapping point pairs to the reference center or the splicing boundary, and use the distances as the weight basis to obtain the weights of each point in the overlapping point pairs

[0115] Calculate the weighted average coordinates of the overlapping point pairs according to the weights of each point in the overlapping point pairs, satisfying the relationship:

[0116]

[0117] where W A and WB respectively represent the weights of the points in the first point cloud data and the second point cloud data in the overlapping point pair, P A and P B respectively represent the coordinates of the points in the first point cloud data and the second point cloud data in the overlapping point pair, d A and d B respectively represent the distances from the points in the first point cloud data and the second point cloud data in the overlapping point pair to the reference center or the stitching boundary;

[0118] Retain the non - overlapping points in the first point cloud data and the second point cloud data, and replace the overlapping point pairs with weighted average coordinate points to obtain the third point cloud data.

[0119] Specifically, by performing stitching and fusion on the first point cloud data and the second point cloud data, a unified third point cloud data is generated. First, by setting a distance threshold, overlapping point pairs with a point cloud spacing within the threshold in the two point cloud data are screened out. Then, the distance from each overlapping point to the reference center or the stitching boundary is calculated, and based on this distance as the weight basis, the weighted average coordinates of each overlapping point pair are calculated. Finally, the non - overlapping points are directly retained, and the overlapping point pairs are replaced with weighted average coordinates, thereby obtaining a third point cloud data with smooth transition and consistent density, ensuring the stitching accuracy and naturalness of the transition of the point cloud data, and providing high - quality basic data for subsequent dimension detection.

[0120] In this step, the distance threshold is a key parameter for screening overlapping point pairs, which determines which points are considered overlapping; the reference center or the stitching boundary is the reference point or line for calculating weights, usually the centroid of the point cloud or the boundary of the stitching area; the weights W A and W B are calculated based on the distance from the point to the reference center or the stitching boundary. The closer the point is, the higher its weight, which reflects the importance of the point in the stitching process; the weighted average coordinate P merged is calculated by combining the coordinates and weights of the overlapping points in the two point cloud data and is used to replace the original overlapping point pair to achieve a smooth transition. These parameters work together to ensure the accuracy and consistency of the point cloud data during the stitching process. Finally, the overlapping parts of the two point clouds are merged, and redundant points are removed, and the complete stitched point cloud (the third point cloud data) is output, providing a complete workpiece model under a unified coordinate system.

[0121] Preferably, the selection of key points in the third point cloud data includes:

[0122] Traverse and calculate the gradients of each point in the third point cloud data in the X, Y, and Z directions, and calculate the structure tensor of each point based on the gradients of each point in the X, Y, and Z directions;

[0123] Calculate the Harris response value based on the structure tensor of each point, set the Harris response threshold. If the Harris response value of a certain point is greater than the Harris response threshold and its Harris response value is the maximum in the neighborhood, then this point is a corner point;

[0124] Analyze the eigenvalues of the structure tensor of each point. If there is a certain point whose first eigenvalue is much larger than the second and third eigenvalues, and the difference between the second and third eigenvalues is less than the preset difference threshold, then this point is regarded as an edge point;

[0125] If the workpiece to be detected is of a symmetric shape, then calculate the centroid in the third point cloud data to determine the center point.

[0126] Specifically, key points are selected by analyzing the geometric features in the third point cloud data, including corner points, edge points and center points. First, calculate the gradients of each point in the X, Y, and Z directions to construct the structure tensor of each point. Then calculate the Harris response value using the structure tensor, and filter out the corner points by setting the response threshold. In addition, edge points are identified by analyzing the eigenvalues of the structure tensor. For workpieces with symmetric shapes, the center is determined by calculating the centroid of the point cloud, which can accurately extract the key feature points in the point cloud data and provide important reference points for subsequent dimension detection. Taking 3D point cloud as an example, the gradients in the x, y, and z directions are calculated using the idea of the Sobel filter, and the gradient calculation formulas in the gradient directions of x, y, and z are as follows:

[0127]

[0128] Among them, represents the convolution operation, I x 、I y 、I z represent the projected images of the point cloud in the x, y, and z directions.

[0129] The structure tensor of each point is a 3×3 matrix, which can be defined as:

[0130]

[0131] Among them, W represents a window centered on each point cloud, which can be a cubic window.

[0132] Subsequently, calculate the Harris response value, that is, Harris response value = det(H) - k(tr(H)) 2 , where det(H) is the determinant of the structure tensor H, tr(H) is the trace of H, and k is an empirical constant, and its value can be between 0.04 - 0.06.

[0133] When determining the edge point, λ 1 、λ2 and λ 3 represent the first eigenvalue, the second eigenvalue, and the third eigenvalue respectively. Assume the eigenvalues of the structure tensor H are λ 1 ≥ λ 2 ≥ λ 3 . If λ 1 >> λ 2 ≈ λ 3 , then this point is identified as an edge point.

[0134] In this step, the gradient refers to the rate of change of each point in the point cloud data in the X, Y, and Z directions, which is used to describe the local geometric features of the point. The structure tensor is a matrix based on gradient information and is used to quantify the local geometric structure of the point. The Harris response value is calculated through the structure tensor and is used to determine whether a point is a corner point. A corner point is a point with significant changes in the local geometric structure and is used for feature matching and dimension measurement. An edge point is a point with significant changes along a certain direction and represents the contour of an object. The centroid is the geometric center of the point cloud data. For a workpiece with a symmetric shape, the centroid can be used as the center point for dimension measurement.

[0135] Preferably, fitting a straight line within the straight line measurement area includes:

[0136] Set the parametric equation of the straight line to be fitted where represents a point on the straight line, represents the direction vector of the straight line;

[0137] Based on minimizing the sum of the squares of the distances from the points to the straight line in the parametric equation, the parameters of the straight line can be obtained, satisfying the relationship:

[0138]

[0139] where, represents a certain point within the straight line measurement area, represents a point on the straight line, represents the direction vector of the straight line.

[0140] The parametric equation is used to describe the geometric features of the straight line, and the sum of the squares of the distances is an index used to measure the difference between the point cloud data and the fitted straight line. By minimizing this index, the best-fitted straight line can be obtained. Specifically, for each point within the straight line measurement area, calculate its distance to the straight line, and take the sum of the squares of these distances as the objective function. By solving the minimum value of the objective function, the best-fitted straight line parameters and By optimizing the minimization algorithm and improving the computational efficiency, the performance of this step can be further enhanced to make it applicable to point cloud data with different complexities.

[0141] In addition to fitting lines, planes are also fitted. In another embodiment, for a set of points within a specific measurement area that requires plane fitting, the least squares method is also used to fit the plane. Let the plane equation be ax + by + cz + d = 0, then the distance from the point (x i , y i , z i ) to the plane is By minimizing the sum of the squares of the distances from the points to the plane, that is the parameters a, b, c, d of the plane can be obtained.

[0142] When calculating the distance between planes, assume there are two planes ax + by + cz + d 1 = 0 and ax + by + cz + d 2 = 0, and their distance D can be calculated by the following formula: If the two planes are parallel (i.e., where is the normal vector of the plane), then take a point (x 1 , y 0 , z 0 , z 0 ) on the plane ax + by + cz + d 1 = 0, then the distance between the planes is

[0143] When calculating the angle between planes, for two planes a 1 x + b 1 y + c 1 z + d 1 = 0 and a 2 x + b 2 y + c 2 z + d 2 = 0, the included angle θ can be calculated through the included angle of their normal vectors, and the formula is where

[0144] Converting the third point cloud data and the data calculated by the dimension detection module to the same coordinate system includes:

[0145] Set a homogeneous transformation matrix for describing the transformation from the base coordinate system of the robotic arm to the end effector coordinate system of the robotic arm

[0146] Convert the third point cloud data from the camera coordinate system to the end effector coordinate system of the robotic arm, satisfying the relationship:

[0147]

[0148] Among them, P arm is the representation of the third point cloud data in the coordinate system of the end effector of the robotic arm, represents the transformation matrix from the camera coordinate system to the coordinate system of the end effector of the robotic arm, and P cam is the representation of the third point cloud data in the camera coordinate system arm;

[0149] Convert the third point cloud data from the coordinate system of the end effector of the robotic arm to the base coordinate system of the robotic arm, satisfying the relationship:

[0150] Specifically, the homogeneous transformation matrix is a matrix used to describe the transformation between coordinate systems. It contains rotation and translation information. The base coordinate system of the robotic arm is the reference coordinate system of the robotic arm, which can be located at the base of the robotic arm. The coordinate system of the end effector of the robotic arm is the coordinate system located at the end of the robotic arm, used to describe the position and orientation of the end of the robotic arm. The camera coordinate system is the reference coordinate system of the 3D camera, used to describe the position and orientation of the point cloud data acquired by the camera. The transformation matrix is used to convert the point cloud data from the camera coordinate system to the coordinate system of the end effector of the robotic arm, while the transformation matrix is used to convert the point cloud data from the coordinate system of the end effector of the robotic arm to the base coordinate system of the robotic arm. These transformation matrices ensure the accurate conversion of the point cloud data between different coordinate systems.

[0151] Therefore, by using the homogeneous transformation matrix to convert the point cloud data from the camera coordinate system to the coordinate system of the end effector of the robotic arm, and then further to the base coordinate system of the robotic arm, it can ensure that all measurement data are in a unified coordinate system, providing an accurate reference for subsequent dimensional inspection. Secondly, it compensates for the motion error of the robotic arm, and can also improve the accuracy of multi-sensor data fusion, enabling the system to cover a wider area of the workpiece, significantly enhancing the inspection accuracy and coverage. Through precise coordinate transformation, the system can adapt to the measurement requirements of complex geometric shapes, solving the problems of insufficient inspection accuracy and poor adaptability in traditional technologies. Finally, this system not only improves the inspection efficiency, but also enhances the stability and reliability in complex industrial environments.

[0152] In the description of this specification, the description of reference terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0153] Although embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A large workpiece size detection system with multi-sensor data fusion, characterized in that: The large workpiece size detection system includes a three-dimensional camera, a PLC, a position sensor, a mechanical arm, a point cloud stitching module and a size detection module, wherein the three-dimensional camera is installed at the end of the mechanical arm; The position sensor is used to detect the position of the workpiece to be detected, and when the workpiece to be detected enters the detectable area, a trigger signal is sent to the PLC according to the position of the workpiece to be detected; After receiving the trigger signal, the PLC sends a control instruction to the robotic arm; The robot arm is used to receive and analyze the control instruction, and move according to a preset motion trajectory and motion posture. When the three-dimensional camera moves to the to-be-detected part of the workpiece to be detected, the three-dimensional camera obtains the corresponding first point cloud data and second point cloud data from above and below each part of the workpiece respectively; The point cloud stitching module is used to obtain the first and second point cloud data, roughly align the feature points in the first and second point cloud data according to the posture information of the three-dimensional camera when acquiring the point cloud data, and finely align the feature points of the first and second point cloud data after the rough alignment according to the KD-Tree constructed based on the second point cloud data, and stitch and fuse the first and second point cloud data to obtain third point cloud data; The size detection module is used to select key points, straight line measurement areas and plane fitting areas in the third point cloud data, calculate the distance between the key points, fit the straight line in the straight line measurement area, fit the plane in the plane fitting area, and calculate the distance and angle between the planes, convert the third point cloud data and the data calculated by the size detection module into the same coordinate system, and obtain the size of the workpiece to be detected.

2. The large workpiece size detection system according to claim 1, characterized in that: The posture information of the three-dimensional camera includes a rotation matrix and a relative displacement; Roughly aligning the feature points in the first and second point cloud data according to the posture information of the three-dimensional camera when acquiring the point cloud data includes: The relative rotation matrix and relative displacement vector of the second point cloud data relative to the first point cloud data are obtained according to the rotation matrix and relative displacement of the three-dimensional camera, satisfying the relationship: Transform each point in the second point cloud data and roughly align the point cloud data to satisfy the relationship: P B ′ =R BA P B +t BA ; Among them, R BA Represents the relative rotation matrix of the second point cloud data relative to the first point cloud data, t BA Represents the relative displacement vector of the second point cloud data relative to the first point cloud data, Represents the transpose of the rotation matrix of the 3D camera when acquiring the first point cloud data, R B Represents the rotation matrix of the 3D camera when acquiring the second point cloud data, t B Indicates the displacement of the 3D camera when acquiring the second point cloud data, t A It represents the displacement of the 3D camera when acquiring the first point cloud data, P B ′ represents the second point cloud data after rough alignment, P B Represents the second point cloud data before coarse alignment.

3. The large workpiece size detection system according to claim 2, characterized in that: The step of finely aligning the feature points of the roughly aligned first and second point cloud data according to the KD-Tree constructed based on the second point cloud data comprises: Select a coordinate axis as the split axis; Sort the points in the second point cloud data according to the coordinate values ​​of the segmentation axis, select the median point as the segmentation point, and divide the second point cloud data into two left and right subtrees, wherein the left subtree contains all points with coordinate values ​​less than the segmentation point, and the right subtree contains all points with coordinate values ​​greater than or equal to the segmentation point; The above steps are repeated recursively for the left and right subtrees until the number of points in the subtree is less than a preset threshold, thereby obtaining a KD-Tree constructed based on the second point cloud data; Starting from the root node of the KD-Tree, by comparing the coordinate value of the point in the first point cloud data on the segmentation axis with the coordinate value of the segmentation point of the current node, enter the left subtree or the right subtree to continue searching. When reaching a leaf node, the point corresponding to the leaf node is used as the current nearest neighbor point candidate; Go back to the parent node and check whether there is a closer point in another subtree of the current node. If another subtree of the current node intersects with a hypersphere centered on the point and with the distance of the current nearest neighbor candidate as the radius, you need to continue searching in the current subtree and update the nearest neighbor candidate. Repeat the above backtracking and checking steps until the entire KD-Tree is traversed and the nearest neighbor point of the first point cloud data is obtained in the second point cloud data.

4. The large workpiece size detection system according to claim 1, characterized in that: The posture information of the three-dimensional camera includes a rotation matrix and a relative displacement; The step of acquiring the posture information of the three-dimensional camera when acquiring the point cloud data includes: Obtain the centroids of the first point cloud data and the second point cloud data, and subtract the centroids of the first point cloud data and the second point cloud data respectively to obtain de-centroided points; The rotation matrices R and H are calculated based on the de-centroided points of the first point cloud data and the second point cloud data, satisfying the relationship: Perform singular value decomposition on H and obtain H = U∑V T , and then according to R = VU T The rotation matrix R is obtained, and the relative displacement t is calculated according to the rotation matrix R and the centroid of the first and second point cloud data, satisfying the relationship: The obtained rotation matrix R and relative displacement t satisfy the following relationship: Among them, E(R,t) represents the objective function, p Ai Represents the i-th point of the first point cloud data, p Bi represents the i-th point of the second point cloud data, q Ai represents the i-th de-centroided point of the first point cloud data, q Bi represents the i-th de-centroided point of the second point cloud data, and They represent the centroids of the first and second point cloud data respectively, and U and V represent orthogonal matrices.

5. The large workpiece size detection system according to claim 1, characterized in that: The step of combining and fusing the first and second point cloud data to obtain the third point cloud data comprises: Setting a distance threshold, and screening out overlapping point pairs in the first and second point cloud data within the distance threshold according to the search structure; Calculate the distance from each point in the overlapping point pair to the reference center or the stitching boundary, use the distance as the weight basis, and obtain the weight of each point in the overlapping point pair; The weighted average coordinates of the overlapping point pairs are calculated according to the weights of each point in the overlapping point pairs, satisfying the relationship: Among them, W A and W B Respectively represent the weights of the points in the first point cloud data and the second point cloud data in the overlapping point pair, P A and P B Respectively represent the coordinates of the points in the first point cloud data and the second point cloud data in the overlapping point pair, d A and d B Respectively represent the distances from the points in the first point cloud data and the second point cloud data in the overlapping point pair to the reference center or the stitching boundary; The non-overlapping points in the first point cloud data and the second point cloud data are retained, and the overlapping point pairs are replaced with weighted average coordinate points to obtain the third point cloud data.

6. The large workpiece size detection system according to claim 1, characterized in that: The selecting of key points in the third point cloud data includes: Traversing and calculating the gradient of each point in the third point cloud data in the X, Y and Z directions, and calculating the structure tensor of each point based on the gradient of each point in the X, Y and Z directions; The Harris response value is calculated according to the structure tensor of each point, and the Harris response threshold is set. If the Harris response value of a point is greater than the Harris response threshold and its Harris response value is the maximum value in the field, then the point is a corner point; Analyze the eigenvalues ​​of the structure tensor of each point. If there is a point whose first eigenvalue is much larger than the second eigenvalue and the third eigenvalue, and the difference between the second eigenvalue and the third eigenvalue is smaller than a preset difference threshold, then the point is considered an edge point. If the workpiece to be inspected is of a symmetrical shape, the center point is determined by calculating the centroid in the third point cloud data.

7. The large workpiece size detection system according to claim 1, characterized in that: Fitting a straight line within the straight line measurement area includes: Set the parametric equation of the straight line to be fitted in represents a point on the line, represents the direction vector of the line; Based on the parameters in the parametric equation, the sum of the squares of the distances from the point to the line can be minimized, and the parameters of the line can be obtained to satisfy the relationship: in, represents a point within the straight line measurement area, represents a point on the line, A vector representing the direction of the line.

8. The large workpiece size detection system according to claim 1, characterized in that: Converting the third point cloud data and the data calculated by the dimension detection module to the same coordinate system includes: Set up a homogeneous transformation matrix that describes the transformation from the robot base coordinate system to the robot end effector coordinate system The third point cloud data is converted from the camera coordinate system to the robot end effector coordinate system to satisfy the relationship: Among them, P arm is the representation of the third point cloud data in the robot end effector coordinate system. represents the transformation matrix from the camera coordinate system to the robot end effector coordinate system, P cam It is the representation of the third point cloud data in the camera coordinate system arm; The third point cloud data is converted from the robot end effector coordinate system to the robot base coordinate system to satisfy the relationship:

Images