Nuclear power station pipeline welding seam automatic identification and measurement method

Abstract

The invention relates to the technical field of pipeline welding seam identification and measurement, aims to solve the problem of uncontrollable data acquisition quality and the problems of robustness and precision of single-feature welding seam extraction, and provides a nuclear power station pipeline welding seam automatic identification and measurement method which comprises the following steps: accurately acquiring pipeline data under a self-adaptive front view posture; carrying out point cloud data processing on the collected data, and carrying out fitting of a pipeline model after segmenting out part of point cloud of the pipeline; after pipeline modeling is completed, a multi-dimensional data fusion method is adopted for weld joint recognition; and after the welding seam area is back projected back to the three-dimensional point cloud space, three-dimensional point cloud of the welding seam area is obtained, modeling is conducted on the welding seam area in combination with the pipeline model, and finally measurement data of the welding seam are obtained. According to the invention, automatic hand-eye calibration and multi-camera calibration of the guiding system can be realized without manual intervention.

Description

Technical Field

[0001] The invention relates to the technical field of pipeline weld identification and measurement, and in particular to an automatic identification and measurement method for nuclear power plant pipeline welds. Background Art

[0002] The main equipment of nuclear power plants are connected by pipelines. The safety and integrity of pipelines are crucial during the operation of nuclear power plants. Each pipeline is welded by short pipes, and the main weak point of the pipeline is the welded joint. Inspection of pipeline welded joints is a key link in pipeline nondestructive testing during in-service inspection. Pipeline weld inspection is a heavy and repetitive operation. In order to reduce the labor intensity of operators and improve the intelligence level of pipeline weld inspection, the use of pipeline weld inspection technology that uses automated operation robots instead of manual labor has become a key technology in the in-service inspection process.

[0003] At present, the main problems include:

[0004] (1) Data collection quality is uncontrollable

[0005] In actual operation scenarios, when autonomous mobile robots conduct pipeline weld inspections, they need to conduct inspections under conditions where the shooting environment changes, the exposure of structured light photography, and the shooting position and angle do not match the environment. Data collection is easily affected by on-site lighting conditions, shooting distance, and camera viewing angle, making it difficult to effectively control the quality of data collection.

[0006] (2) Robustness and accuracy issues of weld extraction based on single feature

[0007] In nuclear power environments, pipeline surfaces are painted or rusted, and the surface textures and characteristics of pipelines are different. The traditional recognition and measurement of pipeline welds using image grayscale or texture features is difficult to apply to all situations and has low accuracy. When the geometric features that distinguish pipeline welds from normal surfaces are used to identify weld areas, the point cloud data shooting angle, background, noise, and point cloud sparseness will interfere, resulting in greater processing difficulty and robustness problems when relying solely on geometric feature processing. Summary of the invention

[0008] The purpose of the present invention is to provide a method for automated identification and measurement of nuclear power plant pipeline welds, to solve the problem of uncontrollable data acquisition quality, as well as the robustness and accuracy of single feature weld extraction. The method is suitable for pipeline weld penetration inspection operations of autonomous mobile robots in nuclear power environments.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A method for automatically identifying and measuring welds in nuclear power plant pipelines, comprising:

[0011] Step 1: Use the camera to collect 2D image data and 3D point cloud data in the scene, identify the pipeline point cloud in the scene through point cloud data processing, fit the pipeline position and orientation, and realize accurate pipeline data collection in an adaptive front-view posture by controlling the robotic arm where the camera is located;

[0012] Step 2: Process the collected data into point cloud data, segment the point cloud of the pipeline part, and then fit the pipeline model;

[0013] Step 3: After the pipeline modeling is completed, the multi-dimensional data fusion method is used to identify the weld. The pipeline modeling is to fit the cylindrical model of the pipeline with the point cloud data, obtain the three parameters of the cylindrical axis direction, the cylindrical radius and the center point, and obtain its three-dimensional geometric model. The multi-dimensional data fusion method is to fuse the two-dimensional color image obtained by the 3D industrial camera and the mapping map of the geometric height features of the three-dimensional point cloud height feature mapping of the cylinder surface to extract the weld.

[0014] Step 4: After back-projecting the weld area back to the three-dimensional point cloud space, the three-dimensional point cloud of the weld area is obtained, and the weld area is modeled in combination with the pipeline model to finally obtain the measurement data of the weld. The weld area modeling is to measure the width, height, and corresponding cylindrical area (center point, radius, and axial parameters) of the weld.

[0015] In some embodiments, after adjusting the environmental exposure, the current field of view state is determined by using the collected environmental point cloud, by detecting whether there is a cylindrical model of the pipeline in the collected three-dimensional point cloud data;

[0016] If there is no cylinder, determine whether the camera's viewing angle is too high or too low based on the current posture;

[0017] If a cylindrical model is detected, the position of the robot arm is adjusted according to the calculated cylindrical model parameters so that the camera is a certain distance away from the surface of the cylinder (the optimal shooting distance of the camera is 400mm-800mm, so when it is necessary to shoot the surface data of the cylinder, the distance needs to be adjusted within 400-800mm to ensure the accuracy of the collected data), and data is collected facing the cylinder.

[0018] In some embodiments, after it is determined that there is a cylindrical model, the position of the end of the robotic arm when facing the cylinder is calculated based on the central axis, center point and orientation information calculated from the cylindrical model and the hand-eye calibration results, and then the robotic arm is controlled; when there is no cylindrical model in the scene, the camera direction is adjusted upward or downward based on the current end posture, and then the data is captured for detection and judgment.

[0019] In some embodiments, the cylinder fitting step includes:

[0020] calculate Where Y i represents the original point cloud, A represents the center point of the point cloud (average);

[0021] Calculate X for all points i i =Y i -A; where X i Indicates the offset value of all points to the center point;

[0022] calculate In the formula, x i ,y i 、z i For X i The three-dimensional coordinate value of , μ and δ represent the intermediate process quantities of the calculation;

[0023] calculate In the formula, F 0 、F 1 、F 2 Indicates the intermediate process quantity of the calculation;

[0024] For each assumed direction W, the following minimization equation is calculated:

[0025] W = [cos(U)sin(V), sin(U)sin(V), cos(V)], angles U and V take three values: [0,0], [π / 2,0], [π / 2,π / 2];

[0026] Calculate the projection matrix P = I-WW T , obtain the skew-symmetric matrix S; where I is the unit matrix, W is the assumed direction vector, P is the calculated projection matrix, and S is the skew-symmetric matrix obtained from P;

[0027] Calculate A = PF 0 P, In the formula, A, is the intermediate process quantity;

[0028] calculate In the formula, Q is the intermediate process quantity;

[0029] Save the upper triangular data p of the matrix P;

[0030] Calculate α = F 0 p, β = Qα; where α and β are intermediate process quantities;

[0031] Calculate G = p T F 2 p-4α T β+4β T F 0β, minimize G, obtain the angles U and V corresponding to the minimum value of G, and then find the cylinder direction vector W; where G is the process quantity, and the other symbols are indicated in the above calculation process;

[0032] The corresponding cylinder center satisfies PC = β, and C is calculated, where C is the coordinate of the cylinder center point;

[0033] The square of the corresponding radius The radius r is obtained; where P is the projection matrix, C is the center point of the cylinder, and X i is the original point cloud;

[0034] In some embodiments, after the pipeline model is fitted, the weld area is extracted by first extracting geometric features from the point cloud data, and obtaining the height of the surface point cloud by calculating the distance from the point cloud to the axis of the pipeline model;

[0035] When processing a convex weld, the height value calculated by the point cloud in the weld area is greater than the radius of the pipe model; when processing a concave weld, the height value calculated by the point cloud in the weld area is less than the radius of the pipe model. Among them, equal means that it is on the surface of the cylinder and is a pipe surface point cloud, not a weld.

[0036] Map the 3D point cloud to the image coordinate system, project the 3D points back to the camera plane, obtain the image coordinates corresponding to the point cloud, map the calculated height value to the grayscale level 0-255, and use the pipeline radius mapping to obtain the 2D image Φ of the 3D point cloud mapping; at the same time, generate a template image of the mapping area The original image is grayscale processed to obtain a grayscale image I. The height map and the grayscale image are fused by weighted fusion to obtain a fused image. as follows:

[0037]

[0038] In the formula, represents the fused image, I represents the grayscale image, Φ represents the two-dimensional image, and α and γ represent the fusion weight values.

[0039] In some embodiments, a fusion image is obtained. ) and then the image (fused image ) is used for denoising and smoothing, and then binarization is performed. The connected region extraction method is used to extract the connected regions obtained in the binary image. For each connected region, the prior information of the weld is used for screening. After the candidate region of the weld in the two-dimensional image is obtained, the candidate region is projected back to the three-dimensional space to obtain the candidate region in the three-dimensional space.

[0040] In some embodiments, the original point cloud is converted to a cylindrical coordinate system, that is, the center point C of the pipe cylinder obtained by fitting is taken as the origin, the central axis direction is the Z axis, the X axis is pointing outward from the center direction of the camera, and the Y axis is perpendicular to the positive direction of the Z and X axes. Fig.12 As shown, the converted 3D point cloud of the weld area Calculate the weld height H s , weld width W s , the center of the circle where the weld is located C s and radius r s , the calculation formula is as follows:

[0041]

[0042] In the formula, represents the three-dimensional point cloud of the weld area, r represents the radius of the fitted pipe cylinder, and [:,2] represents the third column, which is the point cloud The Z-axis component of the point cloud; [:,0] represents the first column. The X-axis component of the point cloud The Y-axis component of .

[0043] In some embodiments, the weld is modeled in the pipeline cylindrical coordinate system to obtain a corresponding cylindrical model of the weld area, and the obtained weld model is converted to the camera coordinate system according to the conversion relationship between the pipeline coordinate system and the camera coordinate system to obtain the original point cloud data coordinate system.

[0044] In some embodiments, the weld model (weld height H s , weld width W s , the center of the circle where the weld is located C s and radius r s Four geometric parameters and a pipe cylinder can construct a geometric model of the weld) to generate a weld model guidance trajectory, and use the trajectory point coordinates to guide the robot arm to perform tracking operations.

[0045] In some embodiments, since the robot arm adopts the eye-on-hand guidance mode during operation, and the camera is fixed relative to the end of the robot arm, the following conversion relationship exists between the robot arm and the camera coordinate system:

[0046]

[0047] In the formula, Represents the transformation matrix from the robot base to the robot end coordinate system, is the transformation matrix from the end of the robot to the camera coordinate system, is the transformation matrix from the camera coordinate system to the object coordinate system, It is the transformation matrix from the robot base coordinate system to the object coordinate system;

[0048] When a camera shoots the same target twice in different poses, and the target is fixed relative to the base of the robot arm, the following equation is obtained:

[0049]

[0050] Since the eyes are in the hand guidance mode The following equation is obtained:

[0051] A*X=X*B

[0052]

[0053] By moving the end of the robotic arm, the same calibration plate is photographed at different positions to obtain a series of parameter A and B values. The Tsai algorithm is used to solve the corresponding camera relative to the robotic arm end coordinate system transformation matrix

[0054] Get the rectangular coordinate X of the weld tracking st After that, convert to the camera coordinate system Then, according to the TCP pose of the end of the robot arm, the coordinate transformation matrix of the end of the robot arm relative to the base of the robot arm can be directly obtained. Get the weld tracking coordinates in the base coordinate system The working posture of the robot arm is pre-set according to the tool head and working mode installed at the end operation, and the tracking operation is performed. The weld is converted to the polar coordinate system (Formula 9), the radius and the starting angle of the tracking are determined in the polar coordinate system, and the polar coordinates of the tracking trajectory are obtained by difference. Then, the generated polar coordinates of the trajectory are converted to rectangular coordinates according to Formula 10 to obtain X st .

[0055] Compared with the prior art, the method for automatic identification and measurement of nuclear power plant pipeline welds provided by the present invention has the following beneficial effects:

[0056] The present invention can replace the existing remote control operation mode of the plugging robot and is used to automatically guide the plugging robot to install and remove the plugging. In addition, the present invention can realize the automatic hand-eye calibration and multi-camera calibration of the guidance system without human intervention.

[0057] Furthermore, the target detection and tracking algorithm based on deep learning can provide a highly robust target tracking effect under visible light shooting conditions. The single-camera independent pose estimation and multi-camera pose fusion methods are adopted. On the one hand, the system has the advantages of being flexible and easy to use in dark-light conversion like a single-camera system. On the other hand, the three-dimensional pose estimation accuracy of multiple cameras is also achieved, which makes the present invention have higher accuracy, applicability and scalability.

[0058] In response to the problem of uncontrollable data acquisition quality, the present invention provides an adaptive data acquisition method, which adaptively adjusts exposure parameters according to grayscale statistical information obtained from images, uses a column detection method to find a pipeline model in the initial shooting data, and adaptively adjusts the camera shooting distance and ensures a frontal shooting angle perpendicular to the pipeline according to the parameters of the pipeline model, thereby improving the quality of data acquisition.

[0059] The present invention aims at the robustness and accuracy problems of weld extraction based on single features, and provides a weld recognition method based on multi-dimensional feature fusion, that is, fusing two-dimensional image features with three-dimensional geometric features to extract weld areas. According to the point cloud data acquired by the structured light camera, a robust cylinder segmentation detection algorithm is used to detect the pipeline model, extract accurate pipeline surface geometric features, and use the structured light camera model to project the geometric features into a two-dimensional image coordinate system, fuse the two-dimensional image grayscale texture features, and construct a multi-dimensional feature response map of the fusion of three-dimensional geometric features and two-dimensional image features. A robust image processing algorithm is used to extract the highlighted weld area, which is back-projected into a three-dimensional point cloud space to obtain a weld area point cloud, and further measure and model the weld area.

[0060] Data collection requires that the pipeline is within the appropriate range of the camera's field of view, faces the camera, and is within the appropriate range of the camera's shooting to ensure sufficient exposure to obtain accurate data. This part adopts an adaptive data collection method; the traditional method based on a single feature source, such as extracting welds based only on color images or grayscale images, or extracting welds based only on geometric features, will be affected by the inherent defects of a single data source, resulting in problems with the robustness and accuracy of weld extraction. The present invention fuses the texture color information of the image and the three-dimensional geometric information extracted from the three-dimensional point cloud to extract the weld, effectively fusing the advantages of the two and making up for their shortcomings, thereby improving the robustness and accuracy of weld extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for the technical description.

[0062] Figure 1 A flow chart of the method for automatic identification and measurement of nuclear power plant pipeline welds provided by the present invention;

[0063] Figure 2 The overall algorithm flow chart provided by the present invention;

[0064] Figure 3 The overall software system logic block diagram provided by the present invention;

[0065] Figure 4 The adaptive acquisition adjustment flow chart provided by the present invention;

[0066] Figure 5 A flow chart of the column detection algorithm provided by the present invention;

[0067] Figure 6 A schematic diagram of column model parameters provided by the present invention;

[0068] Figure 7 A flow chart of the weld area extraction algorithm provided by the present invention;

[0069] Figure 8 A schematic diagram of a feature-fused grayscale image provided by the present invention;

[0070] Fig. 9 A schematic diagram of the weld area extraction result provided by the present invention;

[0071] Fig.10 A schematic diagram of a pipeline coordinate system provided by the present invention;

[0072] Fig.11 A three-dimensional point cloud display of the weld area in the pipeline coordinate system provided by the present invention;

[0073] Fig.12 This is a display diagram of the weld model provided by the present invention. DETAILED DESCRIPTION

[0074] The following is further explained in detail through specific implementation methods.

[0075] For ease of understanding, a brief introduction to pipeline welds is first given. The present invention performs automated identification and measurement of pipeline welds proposed in nuclear power scenarios. Since the identification object is a pipeline weld, the weld type is for an annular weld for straight-through pipeline butt joints. There are usually two types of weld areas: convex and concave, that is, the weld area is higher than the pipeline surface and the weld area is lower than the pipeline surface.

[0076] Usually, in a nuclear power environment, convex welds are the most common and typical pipeline welds. The present invention takes convex welds as an example to illustrate the specific processing process of the method. The processing of concave welds is similar, and reverse processing will be performed when mapping the height response diagram.

[0077] Usually, the color and texture of the pipeline surface will change with the ambient lighting conditions, the surface characteristics of the pipeline such as painting, rust, wear, etc., and the features on the image may vary greatly. It is difficult to use the image directly obtained by the camera to identify and measure the weld area in many scenarios. However, from the geometric features of this type of pipeline annular weld, it can be clearly seen that there will be convex or concave features compared to the normal pipeline surface. Therefore, the structured light camera used in the present invention simultaneously obtains the three-dimensional point cloud, image and depth data of the scene, extracts the geometric features of the scene point cloud based on the data analysis method of the three-dimensional point cloud, and extracts the weld area by using the difference in geometric features between the weld area and the pipeline surface.

[0078] However, in traditional methods, directly processing geometric features in discrete three-dimensional point cloud space is very susceptible to segmentation accuracy problems caused by noise interference, and the algorithm has low processing efficiency. Therefore, the present invention combines the three-dimensional point cloud with geometric features with the intrinsic parameter model of the structured light camera, back-projects it to the two-dimensional image plane, maps the geometric features to the color grayscale space, and uses weighted fusion to fuse part of the color problem information when the color features are significant to form a feature response map, and utilizes the rich image processing methods in image processing to achieve accurate and efficient weld area recognition functions. Finally, it is back-projected back to the point cloud data, or a weld area point cloud is obtained. The weld is modeled according to the weld area point cloud to obtain the corresponding weld height, width, position, model parameters and other information. The overall algorithm framework is as follows Figure 2 As shown, the corresponding logic block diagram of the entire software system is as follows Figure 3 shown.

[0079] like Figure 1 As shown, the present invention provides a method for automatically identifying and measuring nuclear power plant pipeline welds, comprising the following steps:

[0080] Step 1: Start adaptive data collection. The camera (3D industrial camera) collects 2D image data and 3D point cloud data in the scene, and adjusts the adaptive exposure parameters through image grayscale statistics; through point cloud data processing, the pipeline point cloud in the scene is identified, and the rough pipeline position and orientation are fitted. By controlling the robotic arm where the camera is located, accurate pipeline data collection in an adaptive front-facing posture is achieved.

[0081] Step 2: Use the data collected in the normal view posture to process the point cloud data, including point cloud denoising, thinning, and point cloud segmentation. After segmenting the pipeline point cloud, fit the pipeline model.

[0082] Step 3: After the pipeline modeling is completed, the multi-dimensional data fusion method is used to identify the weld. Through the weld model (after fitting the three-dimensional cylindrical model of the pipeline using the point cloud data, further extract the weld area point cloud, calculate the height, width, center point and radius of the weld, and obtain the geometric model parameters of the weld, that is, the weld model), invert the weld surface point cloud based on the height response map of the weld surface model. At this time, the weld area is at a high distance from the weld surface. At the same time, the two-dimensional grayscale image is fused to form a response map of geometric features and grayscale features. A continuous highlight area will appear in the response map, and the weld area is extracted by segmenting the continuous highlight area.

[0083] Step 4: After back-projecting the weld area back into the 3D point cloud space, the 3D point cloud of the weld area is obtained. The weld area is modeled in combination with the pipeline model, and finally the accurate height, width, position and orientation measurement data of the weld are obtained. This generates the tracking track of the weld and guides the robot arm to perform subsequent inspection and repair operations.

[0084] The above steps are described in detail below:

[0085] (1) Adaptive data collection

[0086] In the conventional pipeline weld identification process, since the present invention adopts an automated inspection mode, the robot platform can only move to the designated point to be inspected in advance, which is generally the starting point of the pipeline inspection, so that the camera shooting angle and distance are pre-adjusted, and it is usually difficult to ensure the quality of data collection. The optimal shooting distance of the structured light camera used is between 400mm-800mm, and its acquisition format is limited. It is necessary to adjust the robot arm to between 400mm-800mm away from the pipeline to ensure a certain posture shooting (shooting perpendicular to the pipeline surface). After adjusting the camera shooting posture, since the camera shooting involves exposure adjustment, the choice of exposure usually varies according to the ambient light and the reflection of light on the surface of the object. Too high or too low exposure will cause the point cloud value of the overexposed area to be empty or have a large error, so it is necessary to adjust the exposure.

[0087] Usually, the exposure is adjusted by the weighted value of the grayscale value mean of the acquired image and the proportion of the effective point cloud. When the weights of the two are greater than a certain threshold, the exposure is gradually reduced, otherwise the exposure is gradually increased until the brightness mean is within an appropriate range.

[0088] After adjusting the camera exposure, the current field of view state is determined by the collected environmental point cloud. The collected 3D point cloud data is used to detect whether there is a cylindrical model of the pipeline. If there is no cylinder, the camera's viewing angle is determined to be too high or too low according to the current posture. If a cylindrical model is detected, the position of the robot arm is adjusted according to the calculated cylindrical model parameters so that the camera is 450 mm (for example) away from the surface of the cylinder and data is collected directly on the cylinder. The flow chart is as follows: Figure 4 shown.

[0089] Among them, the cylinder detection is mainly carried out by cylinder fitting, and whether it is a cylinder model is comprehensively judged by the error of the fitting result and the number of point clouds included. After judging that there is a cylinder model, according to the central axis, center point and orientation information calculated by the cylinder model, the position of the end of the robot arm facing the cylinder at a distance of 450mm from the cylinder surface can be calculated according to the hand-eye calibration results, and then the robot arm can be controlled; when there is no cylinder model in the scene, the camera direction is adjusted upward or downward according to the current end posture, and then the data is captured for detection and judgment. Therefore, the main module of the adaptive acquisition stage is the detection of cylinders. The following is a detailed description of the cylinder detection method.

[0090] like Figure 5 As shown, the steps of column detection include: initial point cloud, point cloud preprocessing, point cloud segmentation, column fitting, and column judgment.

[0091] Among them, point cloud preprocessing mainly includes point cloud filtering and point cloud sampling. Point cloud filtering mainly filters out invalid point clouds and filters out point clouds with a Z-direction distance greater than 1.5m or less than 0.001m. Point cloud sampling uses uniform sampling of detection to obtain a certain number of points from the filtered point cloud data. 6000 points are set in the present invention. That is, the sampling interval is obtained by dividing the total number of point clouds by the number of samples. Since the point cloud output by the structured light camera is a point cloud arranged continuously in rows and columns, a uniform sampling result can be obtained by directly using a sampling method with a fixed index interval, which can greatly ensure the sampling efficiency.

[0092] After the preprocessing is completed, the point cloud is segmented. The point cloud segmentation uses the DBSCAN clustering algorithm to segment the point cloud into clusters of point cloud sets, and then fit the cylinder model on each point cloud set.

[0093] An infinite cylinder is defined by a center point C specified by an axis containing a point, a unit direction vector W, and a radius r, as Figure 6 As shown. Two vectors U and V of unit length, {U, V, W} is a right-handed orthogonal set; that is, all vectors are of unit length, perpendicular to each other, and U×V=W, V×W=U, W×U=V. Then any point X on the cylinder can be uniquely represented as:

[0094] X=C+y 0 U+y 1 V+y 2 W=C+RY (1)

[0095] Where R is the rotation matrix, whose columns are U, V, and W; Y is the column vector, whose values ​​are y 0 ,y 1 and 2 . And the radius |y2 |=|W·(XC)|≤h / 2, where h is the height of the cylinder. Then for the input point cloud Y i , the algorithm steps of cylinder fitting are as follows:

[0096] calculate Where Y i represents the original point cloud, A represents the center point of the point cloud (average);

[0097] Calculate X for all points i i =Y i -A; where X i Indicates the offset value of all points to the center point;

[0098] calculate In the formula, x i ,y i 、z i For X i The three-dimensional coordinate value of , μ and δ represent the intermediate process quantities of the calculation;

[0099] calculate In the formula, F 0 、F 1 、F 2 Indicates the intermediate process quantity of the calculation;

[0100] For each assumed direction W, the following minimization equation is calculated:

[0101] W = [cos(U)sin(V), sin(U)sin(V), cos(V)], angles U and V take three values: [0,0], [π / 2,0], [π / 2,π / 2];

[0102] Calculate the projection matrix P = I-WW T , obtain the skew-symmetric matrix S; where I is the unit matrix, W is the assumed direction vector, P is the calculated projection matrix, and S is the skew-symmetric matrix obtained from P;

[0103] Calculate A = PF 0 P, In the formula, A, is the intermediate process quantity;

[0104] calculate In the formula, Q is the intermediate process quantity;

[0105] Save the upper triangular data p of the matrix P;

[0106] Calculate β = F 0 p, β = Qα; where α and β are intermediate process quantities;

[0107] Calculate G = pT F 2 p-4α T β+4β T F 0 β, minimize G, obtain the corresponding angles U and V when G takes the minimum value, and then find the cylinder direction vector W; where G is the process quantity;

[0108] The corresponding cylinder center satisfies PC = β, and the coordinates C of the center point of the cylinder can be calculated;

[0109] The square of the corresponding radius Solve to calculate the radius r; where P is the projection matrix, C is the center point of the cylinder, and X i is the original point cloud;

[0110] After the cylinder model parameters are fitted, calculate the distance from the collection point to the cylinder axis W Assuming that the number of filtered point clouds is N, and the number of current fitting set point clouds is n, the model loss value after fitting is:

[0111]

[0112] Where B represents the model loss value after fitting, R represents the distance, r represents the radius, N represents the number of point clouds after filtering, and n represents the number of point clouds in the current fitting set.

[0113] In formula (2), the ratio of the average fitting error to the set proportion is used as the loss value. The larger the error and the lower the model proportion, the less credible the column model is. By setting the loss value threshold, it is determined whether it meets the expected column model.

[0114] The distance between the camera and the axis of the cylinder can be calculated through the cylinder model, that is, the distance from the origin to W. After subtracting the radius of the cylinder, the distance between the camera and the surface of the cylinder can be obtained. The cylinder direction W is the direction of the pipeline, and the line connecting the origin to the foot of the perpendicular to the axis W points to the direction of the foot of the perpendicular, which is the posture of the camera when shooting. In turn, the position and posture of the camera when shooting can be determined.

[0115] (2) Pipe cylinder fitting

[0116] The pipeline cylinder fitting method is similar to the cylinder detection method in the above-mentioned adaptive data acquisition. It is necessary to eliminate the interference of background and noise in the point cloud data, detect and segment the point cloud of the pipeline part, and use it to fit the pipeline cylinder model. The specific method flow is similar to the main detection method flow, so it will not be repeated here.

[0117] (3) Weld area extraction

[0118] In the process of adaptive data acquisition, the camera exposure is first adjusted by the grayscale mean of multiple image data and the proportion of effective point clouds. The distance and posture of camera data acquisition are adjusted according to the cylinder model detected in the point cloud data, so that the camera can shoot data at a distance of 450mm from the cylinder and in a straight-on posture. After data acquisition, the point cloud data processing stage is entered. Similar to the point cloud preprocessing process in the above-mentioned cylinder detection, the edge point cloud will be filtered out during the point cloud filtering process. This is mainly because the edge point cloud has more noise. The point cloud data corresponding to 20-30 pixels on the edge will be selected in the image coordinate space for filtering to eliminate the interference of edge points. The subsequent processing is exactly the same.

[0119] The pipeline cylinder model fitting algorithm also adopts the cylinder fitting algorithm in the above cylinder detection. In order to eliminate the interference of external points as much as possible, the RANSAC algorithm is used to screen out internal points for accurate model fitting.

[0120] After the pipeline model is fitted, the weld area extraction phase begins. The weld area extraction first extracts geometric features from the point cloud data. Since the annular weld targeted by the present invention will present convex or concave geometric features relative to the pipeline surface, the height of the surface point cloud can be obtained by calculating the distance from the point cloud to the pipeline model axis. When processing a convex weld, the height value calculated by the point cloud of the weld area will be greater than the radius of the pipeline model; when processing a concave weld, it must be lower than the pipeline radius. Therefore, the height value of the point cloud can be calculated as its geometric feature. Using the geometric model and camera intrinsic parameters of the structured light camera, the three-dimensional point cloud can be mapped to the image coordinate system, as shown in formula (3):

[0121]

[0122] Where [u,v] is the pixel coordinate, K is the camera intrinsic parameter matrix, [R|t] is the required pose matrix (extrinsic parameter matrix), when the camera center is the origin of the three-dimensional coordinate space, R is the unit matrix, t is the zero vector, and [x,y,z] is the three-dimensional point cloud coordinate.

[0123] Projecting the 3D points back to the camera plane, we can get the image coordinates corresponding to the point cloud, map the calculated height value to the grayscale level 0-255, and use the pipeline radius mapping for the pixel value of each 3D point pair to get the 2D image Φ of the 3D point cloud mapping. At the same time, generate a template image of the mapping area Represents the valid 3D point cloud area. The original image is grayscale processed to obtain a grayscale image I. The height map and the grayscale image are fused by weighted fusion, as shown in formula (4), to obtain the fused image

[0124] In the formula, represents the fused image, I represents the grayscale image, Φ represents the two-dimensional image, α represents the grayscale image weight value, and γ represents the height map weight value.

[0125] Usually, the significance of height features is higher than that of grayscale features, and grayscale features may vary due to changes in lighting conditions and surface characteristics, so the weight value of grayscale features is usually lower, while the weight of geometric features is set higher.

[0126] Get the fused grayscale image (such as Figure 8 After that, the image processing algorithm can be used to extract the weld corresponding to the highlighted area with significant features. The process of extracting the weld area from the image is as follows: Figure 7 shown.

[0127] After obtaining the fusion image, the image needs to be denoised and smoothed. First, the median filter is used to remove the interference of stray noise points in the image, and then the Gaussian filter is used for smoothing. Since there will be certain holes and discontinuities in the image after 3D point mapping, the morphological closing operation is used at this time to further process the image, mainly to fill the discontinuous areas with holes. The mask image will also be processed consistently in this process.

[0128] After the image is smoothed, it is binarized to extract the candidate weld area with a higher height. Since the edge point cloud has a poor accuracy during the 3D point cloud generation process, it will cause great interference to the binarized result. At this time, the mask image that previously excluded the edge is used for mask filtering to eliminate the interference of the edge area. At the same time, morphological opening operations are performed to eliminate the interference of some small area bright spots and obtain larger candidate areas in continuous blocks.

[0129] In the region extraction process, the connected region extraction method is first used to extract the connected regions obtained in the binary image. For each connected region, the prior information of the weld is used for screening. The present invention uses the area and aspect ratio information of the weld for screening, and the regions with aspect ratio values ​​greater than 3 and areas greater than 1000 are finally output as candidate regions.

[0130] After obtaining the candidate area of ​​the weld in the two-dimensional image, the candidate area is further projected back to the three-dimensional space to obtain the candidate area in the three-dimensional space. Then the three-dimensional geometric features of the weld are screened in the three-dimensional space. The screening criteria include: the direction of the quilting is perpendicular to the central axis of the cylinder, the arc of the ring where the weld is located is greater than 20 degrees, and the width of the weld is less than 100mm. Finally, the accurate weld is screened out. The result example is as follows Fig. 9 shown.

[0131] (4) Pipeline weld modeling

[0132] After the pipeline weld area is extracted, it is necessary to further calculate the height, width, and center circle parameters of the weld, that is, to model the weld, measure and locate the weld, so as to guide the robot to perform corresponding operations.

[0133] In order to facilitate the calculation of weld model parameters, the entire point cloud data is first converted into a pipeline coordinate system with the center of the cylinder as the center point, the central axis direction as the Z axis, the X axis pointing outward along the center direction of the camera as the X axis, and the Y axis perpendicular to the positive direction of the Z and X axes as the Y axis, as shown in the figure. Fig.10 shown.

[0134] The coordinate system transformation matrix is ​​calculated by first calculating the rotation matrix, that is, calculating the rotation matrix of the axis direction vector of the pipeline cylinder model to the Z axis [0,0,1]. The translation matrix is ​​-C. The result is as follows Fig.11 shown.

[0135] After the conversion, the corresponding weld height, weld width, center and radius of the circle where the weld is located can be directly calculated from the point cloud. Assume that the three-dimensional point cloud of the weld area in the pipeline cylindrical coordinate system is is an n*3 matrix, n is the number of point clouds in the weld area, and the corresponding weld height H s , weld width W s , the center C of the pipe weld circle in the pipe cylindrical coordinate system s and the corresponding radius r s They are respectively formula (5), as follows:

[0136]

[0137] In the formula, It represents the three-dimensional point cloud of the weld area in the pipe cylindrical coordinate system, and r represents the radius of the cylinder fitted by the radius pipe. [:,2] represents the third column, that is, the point cloud The Z-axis component of the point cloud; [:,0] represents the first column. The X-axis component of the point cloud Y-axis component of

[0138] By modeling the weld in the pipeline cylindrical coordinate system, the corresponding cylindrical model of the weld area can be obtained. According to the conversion relationship between the pipeline coordinate system and the camera coordinate system, the obtained weld model can be converted to the camera coordinate system, which is the original point cloud data coordinate system. The final weld model is obtained, such as Fig.12 shown.

[0139] (5) Hand-eye calibration and weld tracking control

[0140] After obtaining the weld model, the next step is to use the weld model to generate a weld model guidance trajectory, and use the trajectory point coordinates to guide the robot to perform tracking operations. Since the weld model or weld trajectory extracted from the original point cloud is in the camera coordinate system, it needs to be converted to the robot's base coordinate system before it can be used to guide the robot's tracking operation. Therefore, hand-eye calibration is first required to obtain the conversion relationship between the camera coordinate system and the base (robot base) coordinate system.

[0141] Since the robot arm uses the eye-on-hand guidance mode during operation, that is, the camera is fixed relative to the end of the robot arm, there will be the following conversion relationship between the robot arm and the camera coordinate system:

[0142]

[0143] In the formula, Represents the coordinate system transformation matrix from the robot base (Base) to the robot end (End), is the transformation matrix from the end of the robot (End) to the camera coordinate system (Camera), is the conversion matrix from the camera coordinate system (Camera) to the object coordinate system (Object), It is the transformation matrix from the robot base coordinate system (Base) to the object coordinate system (Object).

[0144] When a camera shoots the same target twice in different poses, and the target is fixed relative to the base of the robot arm, the following equation can be obtained:

[0145]

[0146] Since the eyes are in the hand guidance mode The following equation (Formula 8) can be obtained:

[0147] make The above formula is converted into:

[0148] A*X=X*B

[0149] In the formula, A and B represent process parameters, and X is the final desired result.

[0150] By moving the end of the robotic arm and photographing the same calibration plate at different positions, a series of parameter A and B values ​​can be obtained. The Tsai algorithm can be used to calculate the corresponding camera relative to the robotic arm end coordinate system transformation matrix.

[0151] After obtaining the hand-eye calibration results, the next step is to generate the weld tracking coordinates. After obtaining the weld model, for ease of calculation, the calculation is performed in the pipe cylindrical coordinate system. First, obtain the weld center Cs and weld radius r s After that, the weld coordinate points are converted to the polar coordinate system (Formula 9):

[0152]

[0153] After converting to the polar coordinate system, the central angle range Δθ and arc length Δθ*r corresponding to the weld in the entire visible area can be calculated t , where r t is the tracking radius, which is usually larger than the weld radius r s Since the robot control process uses discrete points with a certain interval on a continuous trajectory, the trajectory interval is set to Δ l , then the sampling center angle interval is Δ l / r t , you can get the polar coordinates of the sampling points where i is in [0, 1, 2, ..., Δθ*r t / Δ l After obtaining the trajectory polar coordinates, it can be converted to the pipeline cylinder rectangular coordinate system according to formula 10:

[0154] x=ρ*cos(θ)

[0155] y=ρsin(θ)

[0156] Get the rectangular coordinate X of the weld tracking st After that (polar coordinates of the weld trajectory), convert it to the camera coordinate system Then, according to the TCP pose of the end of the robot arm, the coordinate transformation matrix of the end of the robot arm relative to the base of the robot arm can be directly obtained. The weld tracking coordinates in the base coordinate system can be obtained That is, the most tracking robot arm controls the position point coordinates, and the robot arm's operating posture is pre-set according to the tool head installed at the end operation and the operating mode to perform tracking operations.

[0157] In some embodiments, the present invention utilizes an existing mobile robot to perform automated pipeline weld identification and measurement in a nuclear power environment. After the autonomous mobile robot moves to the point to be detected, the above steps of the present invention are executed.

[0158] This method involves an automated identification and measurement device for nuclear power plant pipeline welds, which is mainly used for automated identification and measurement of welds during in-service inspections in nuclear power plants, paving the way for subsequent automated nondestructive testing and automated weld inspections. The application scenario is mainly to carry a robotic arm for weld penetration inspection on an autonomous mobile robot, and the robot body is equipped with sensors such as laser radar, camera, and IMU for environmental SLAM (simultaneous localization and mapping) mapping, positioning, and navigation. In the initial state, the robot's autonomous mapping technology will be used to build a three-dimensional map of the inspection plant. By manually marking the inspection points of the pipeline to be inspected, the robot moves to the inspection point through autonomous navigation to inspect the pipeline.

[0159] The method proposed in the present invention uses the structured light camera carried by the end of the operating robot arm to automatically identify the weld and mark the weld area according to the captured three-dimensional data of the pipeline after the robot reaches the inspection point, and generates a weld trajectory for use in the robot's subsequent penetration inspection and other operations.

[0160] The hardware components of the device of the present invention include a structured light camera, an industrial computer and a robotic arm that are electrically / communicatively connected, of which only the operating robotic arm and the structured light camera installed on the robotic arm are closely related to the present invention. The robotic arm adopts the UR5e robotic arm, and the camera adopts an existing industrial camera, which uses 465nm blue stripe structured light and advanced phase shift profile algorithm to form a single snapshot array image. Three-dimensional point cloud, RGB color image and depth map data can be obtained simultaneously.

[0161] Accordingly, in the process of weld detection and identification, traditional methods basically focus on the acquired images or 3D point cloud data containing the weld area, and rarely involve the optimization and adjustment of data in the acquisition stage. For example, under autonomous operation conditions, the inspection or identification process of whether the weld or inspection object is in the shooting area, the quality control and optimization of data acquisition, the adjustment of the acquisition position and sensor posture, etc. are rarely mentioned, and this part of the content is the basis and premise of subsequent work and must be guaranteed in autonomous operation scenarios.

[0162] Similar to the traditional processing method based on three-dimensional point cloud, the present invention can directly obtain the three-dimensional point cloud, image and depth data in the scene through the structured light camera, and extract the geometric features of the scene in the three-dimensional space according to the geometric features of the pipeline weld. However, the traditional method directly extracts the weld area in the three-dimensional point cloud space or feature space, which will cause the processing process to be complicated and inefficient due to the discontinuity of the three-dimensional point space or feature space. It is usually difficult to integrate the continuity characteristics of the weld space area, and the processing process is easily disturbed by noise, resulting in poor processing effect. Based on the spatial characteristics of the three-dimensional point cloud, the present invention maps the three-dimensional point cloud containing spatial features to a two-dimensional image according to the internal parameter matrix of the structured light camera, constructs a feature response map by weighted fusion of geometric features and color features, and uses an image processing algorithm to segment the weld area. By projecting the weld area back to the three-dimensional space, the weld area point cloud can be obtained, and then the weld area can be measured and modeled.

[0163] This processing method can effectively integrate the geometric features of the weld and the color and texture features of the two-dimensional image, map the three-dimensional coordinates into an orderly and continuous feature correspondence map, and at the same time combine an accurate and efficient image processing algorithm to segment the weld area and obtain the weld point cloud. The present invention has obvious effects in processing accuracy and robustness.

[0164] The invention relates to an automatic weld identification and measurement device for nuclear power plant pipelines, which is mounted on an autonomous mobile robot platform. The autonomous mobile robot platform maps and plans pipeline inspection points in advance, and when the automatic pipeline inspection is started, it arrives at the pipeline inspection point. The automatic weld identification and measurement device starts to start.

[0165] First, the adaptive data acquisition module will be started, and the structured light sensor will start to collect data. The exposure parameters of data acquisition will be adjusted by calculating the grayscale mean value and the percentage of effective point cloud in the image. When the detected grayscale value and the percentage of effective point cloud reach the specified threshold, the collected point cloud data will be detected by the pipeline column model detection algorithm to see if the collected point cloud contains a pipeline model. When there is no pipeline model, the end posture data of the robot arm will be used to judge and control the camera viewing angle to adjust up and down. When the pipeline model is detected, the end posture of the robot arm control is obtained according to the pipeline model parameters and the hand-eye calibration parameters for calibration, and the end camera is controlled to reach a shooting posture 450mm away from the pipeline surface to collect data.

[0166] Then, the data collected in the ideal posture is first preprocessed into a 3D point cloud, and then the cylindrical model is fitted to obtain the pipeline model, and the height from the point cloud to the central axis of the pipeline is calculated. The 3D point cloud is projected onto the image plane through the camera internal parameters, and the height value is mapped to the pixel value to obtain the height projection map. The height projection map is weightedly fused with the grayscale map to obtain a fused image of geometric features and grayscale features, and the image processing algorithm is used to extract the weld area with significant fusion features.

[0167] Finally, the extracted weld area is projected back into three-dimensional space to obtain the three-dimensional point cloud of the weld. Further, the circular ring model of the weld is obtained based on the cylindrical model of the pipe, and the measurement parameters such as the height, width, and position of the circular ring of the weld are output.

[0168] The output pipeline weld model can further generate a weld arc trajectory, and generate a weld inspection trajectory at the end of the robotic arm based on the hand-eye calibration results, thereby guiding the robotic arm to perform operations such as weld penetration and weld repair.

[0169] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for automatic identification and measurement of nuclear power plant pipeline welds, characterized in that: include: Step 1: Use the camera to collect 2D image data and 3D point cloud data in the scene, identify the pipeline point cloud in the scene through point cloud data processing, fit the pipeline position and orientation, and realize accurate pipeline data collection in an adaptive front-view posture by controlling the robotic arm where the camera is located; Step 2: Process the collected data into point cloud data, segment the point cloud of the pipeline part, and then fit the pipeline model; Step 3: After the pipeline modeling is completed, the multi-dimensional data fusion method is used to identify the weld; Step 4: After back-projecting the weld area back into the 3D point cloud space, the 3D point cloud of the weld area is obtained. The weld area is modeled in combination with the pipeline model to finally obtain the measurement data of the weld.

2. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1 is characterized in that: In step 1, after adjusting the environmental exposure, the current field of view state is determined by the collected environmental point cloud, by detecting whether there is a cylindrical model of the pipeline in the collected three-dimensional point cloud data; If there is no cylinder, determine whether the camera's viewing angle is too high or too low based on the current posture; If a cylindrical model is detected, the position of the robot arm is adjusted according to the calculated cylindrical model parameters so that the camera is at a certain distance from the surface of the cylinder and collects data directly on the cylinder.

3. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1 or 2, characterized in that: When it is determined that there is a column model, the position of the end of the robot arm facing the column is calculated based on the central axis, center point and orientation information calculated by the column model and the hand-eye calibration result, and then the robot arm is controlled; When there is no column model in the scene, the camera direction is adjusted upward or downward based on the current terminal posture, and then the data is captured for detection and judgment.

4. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1, characterized in that: The cylinder fitting steps include: calculate Where Y i represents the original point cloud, and A represents the center point of the point cloud; Calculate X for all points i i =Y i -A; where X i Indicates the offset value of all points to the center point; calculate In the formula, x i ,y i 、z i For X i The three-dimensional coordinate value of , μ and δ represent the intermediate process quantities of the calculation; calculate In the formula, F0, F1, and F2 represent the intermediate process quantities of the calculation; For each assumed direction W, the following minimization equation is calculated: W = [cos(U)sin(V), sin(U)sin(V), cos(V)], angles U and V take three values: [0,0], [π / 2,0], [π / 2,π / 2]; Calculate the projection matrix P = I-WW T , obtain the skew-symmetric matrix S; where I is the unit matrix, W is the assumed direction vector, P is the calculated projection matrix, and S is the skew-symmetric matrix obtained from P; Calculate A = PF0P, In the formula, A, is the intermediate process quantity; calculate In the formula, Q is the intermediate process quantity; Save the upper triangular data p of the matrix P; Calculate α=F0p, β=Qα; where α and β are intermediate process quantities; Calculate G = p T F2p-4α T β+4β T F0β, minimize the value of G, obtain the angles U and V corresponding to the minimum value of G, and then find the cylinder direction vector W; where G is the process quantity; The corresponding cylinder center satisfies PC = β, and the coordinates C of the cylinder center point are calculated; The square of the corresponding radius The radius r is obtained; where P is the projection matrix, C is the center point of the cylinder, and X i is the original point cloud.

5. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1, characterized in that: In step 3, after the pipeline model is fitted, the weld area is extracted. First, the geometric features are extracted from the point cloud data, and the height of the surface point cloud is obtained by calculating the distance from the point cloud to the axis of the pipeline model; When processing a convex weld, the height value calculated by the point cloud in the weld area is greater than the radius of the pipe model; when processing a concave weld, the height value calculated by the point cloud in the weld area is less than the radius of the pipe model. Map the 3D point cloud to the image coordinate system, project the 3D points back to the camera plane, obtain the image coordinates corresponding to the point cloud, map the calculated height value to the grayscale level 0-255, and use the pipeline radius mapping to obtain the 2D image Φ of the 3D point cloud mapping; at the same time, generate a template image of the mapping area The original image is grayscale processed to obtain a grayscale image I. The height map and the grayscale image are fused by weighted fusion to obtain a fused image. as follows: In the formula, represents the fused image, I represents the grayscale image, Φ represents the two-dimensional image, and α and γ represent the fusion weight values.

6. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1 or 5, characterized in that: After obtaining the fused image, the fused image is denoised and smoothed, and then binarized. The connected region extraction method is used to extract the connected regions obtained in the binary image. For each connected region, the prior information of the weld is used for screening. After the candidate region of the weld in the two-dimensional image is obtained, the candidate region is projected back to the three-dimensional space to obtain the candidate region in the three-dimensional space.

7. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1, characterized in that: In step 4, the converted 3D point cloud of the weld area is Calculate the weld height H s , weld width W s , the center of the circle where the weld is located C s and radius r s , the calculation formula is as follows: In the formula, represents the three-dimensional point cloud of the weld area, and r represents the radius of the fitted pipe cylinder.

8. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 7, characterized in that: The weld is modeled in the pipeline cylindrical coordinate system to obtain the corresponding cylindrical model of the weld area. According to the conversion relationship between the pipeline coordinate system and the camera coordinate system, the obtained weld model is converted to the camera coordinate system to obtain the original point cloud data coordinate system.

9. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 8, characterized in that: The weld model is used to generate the weld model guiding trajectory, and the trajectory point coordinates are used to guide the robot arm to perform tracking operations.

10. The method for automatic identification and measurement of nuclear power plant pipeline welds according to claim 1, characterized in that: In step 4, since the robot arm uses the eye-on-hand guidance mode during operation and the camera is fixed relative to the end of the robot arm, the following conversion relationship exists between the robot arm and the camera coordinate system: In the formula, Represents the transformation matrix from the robot base to the robot end coordinate system, is the transformation matrix from the end of the robot to the camera coordinate system, is the transformation matrix from the camera coordinate system to the object coordinate system, It is the transformation matrix from the robot base coordinate system to the object coordinate system; When a camera shoots the same target twice in different poses, and the target is fixed relative to the base of the robot arm, the following equation is obtained: Since the eyes are in the hand guidance mode The following equation is obtained: A*X=X*B By moving the end of the robotic arm, the same calibration plate is photographed at different positions to obtain a series of parameter A and B values. The Tsai algorithm is used to solve the corresponding camera relative to the robotic arm end coordinate system transformation matrix Get the rectangular coordinate X of the weld tracking st After that, convert to the camera coordinate system Then, according to the TCP pose of the end of the robot arm, the coordinate transformation matrix of the end of the robot arm relative to the base of the robot arm can be directly obtained. Get the weld tracking coordinates in the base coordinate system The robot arm's operating posture is pre-set based on the tool head installed at the end operation and the operating mode to perform tracking operations.

Images