Defect quantification method based on fusion of three-dimensional camera and neural network and application thereof

Abstract

The present application relates to the technical field of image processing, and discloses a defect quantification method based on fusion of a three-dimensional camera and a neural network and application thereof.The defect quantification method comprises the following steps: step A, cutting a large-area gray image and point cloud data collected by a three-dimensional camera into a smaller area related to a defect to obtain a smaller-area gray image and point cloud data;step B, performing surface fitting on the point cloud data of the smaller area after cutting to obtain a smooth surface model;step C, processing the smaller-area gray image using an instance segmentation neural network model to obtain a corresponding mask image;step D, extracting sub-point cloud data of a defect edge from the point cloud data of the smaller area according to the defect edge on the mask image;and step E, performing difference calculation on the sub-point cloud data and the smooth surface model to obtain geometric information of the defect.The calculation method is simple, and the calculation efficiency and accuracy are high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a defect quantification method based on fusion of three-dimensional cameras and neural networks and application thereof. BACKGROUND

[0002] As the key power device of an aircraft, aerospace engines are known as the "heart" of aircraft and rockets. Their independent development is of great significance to national defense construction, energy security and environmental sustainability. Among them, the blade, as the core component of aerospace engines that guides airflow and generates power, the detection of its key quality characteristics directly affects the service safety and reliability of the aircraft.

[0003] Among the many components of the engine, the number of blades is large, and their manufacturing quantity accounts for about one-third of the entire engine manufacturing quantity. Blades have many characteristics, such as large number, complex shape, high precision requirement, difficult processing, and frequent failures, and are key components of the engine. The engine relies on numerous blades to complete the compression and expansion of gas to generate powerful power to propel the aircraft forward. Therefore, the quality of the blades is a key factor affecting the energy conversion efficiency and working performance of the engine.

[0004] Currently, the main detection method for blade surface defects is manual visual inspection. From the perspective of detection quality, the detection results are greatly affected by human factors. From the perspective of detection efficiency, the blade appearance is complex, and there are many detection items, so manual detection efficiency is low. From the perspective of detection data, manual detection results are only qualitative judgments, lack of quantitative data, and the results are difficult to store digitally, which is not conducive to later quality problem tracing. From the perspective of detection quantity, there is still a lack of intelligent detection capability for batch blade surface defects on site, and the manual visual inspection method has reached a production capacity bottleneck, which is difficult to meet the needs of production and delivery. SUMMARY

[0005] The main purpose of the present application is to provide an image-based engine blade defect automatic detection method and an engine blade defect detection system, which digitally, automatically and intelligently detects blade surface defects to achieve stable quality, high efficiency, high traceability and quantitative detection results.

[0006] To achieve the above purpose, the technical scheme of the image-based engine blade defect automatic detection method and the engine blade defect detection system provided by the present application is as follows:

[0007] The image-based engine blade defect automatic detection method comprises the steps of: using a robot gripper to hold the blade and move it to a detection site, and using a two-dimensional camera and a three-dimensional camera to take a picture of the blade at the detection site.

[0008] Step (1), a two-dimensional camera is used to take pictures of the blade held by the robot gripper to obtain a plurality of two-dimensional images covering the surface of the blade;

[0009] Step (2), a target detection neural network model is used to detect defects in the plurality of two-dimensional images;

[0010] Step (3), for the two-dimensional images in which defects are detected, three-dimensional coordinates of the defect center in the robot base coordinate system are calculated, and then the gripper is caused to move the defect center to the imaging center of the three-dimensional camera;

[0011] Step (4), a three-dimensional camera is used to collect a grayscale image and point cloud data of the defect;

[0012] Step (5), a defect quantification method based on the fusion of a three-dimensional camera and a neural network is used to process the grayscale image and point cloud data of the defect to obtain geometric information of the defect.

[0013] The engine blade defect detection system comprises a tray for storing blades, wherein a blade storage groove is arranged on the tray; a robot for clamping and moving the blades, wherein the robot has a gripper and a mechanical arm; a two-dimensional camera for taking pictures of the blades held by the gripper; a three-dimensional camera for taking pictures of the blades held by the gripper; and a light supplementing light source for supplementing light when the two-dimensional camera and the three-dimensional camera take pictures.

[0014] In the above image-based engine blade defect automatic detection method, when steps (3) and (5) adopt the following calculation method for obtaining the coordinates of the defect center in the two-dimensional image and the defect quantification method based on the fusion of a three-dimensional camera and a neural network, the calculation method is simple, and the calculation efficiency and accuracy are high.

[0015] Therefore, the present application provides a calculation method for obtaining the coordinates of the defect center in the two-dimensional image and a defect quantification method based on the fusion of a three-dimensional camera and a neural network, and the technical solutions are as follows:

[0016] The calculation method for obtaining the coordinates of the defect center in the two-dimensional image is used to make the gripper of the robot hold the test object and move the defect center of the test object to the imaging center of the three-dimensional camera, and the calculation method comprises the following steps:

[0017] Step 100, multiplying the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image and the intrinsic matrix of the two-dimensional camera to calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system;

[0018] Step 200, calculating the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera;

[0019] Step 300, using the transformation matrix of the two-dimensional camera coordinate system to the robot base coordinate system, the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are converted into three-dimensional coordinates in the robot base coordinate system;

[0020] Step 400, using the transformation matrix of the robot base coordinate system to the robot gripper coordinate system, the three-dimensional coordinates of the defect in the robot base coordinate system are converted into three-dimensional coordinates in the robot gripper coordinate system;

[0021] Step 500, the three-dimensional coordinates of the defect in the robot gripper coordinate system are subtracted from the origin of the robot gripper coordinate system, and a vector representing the relative position of the defect in the robot gripper coordinate system is calculated;

[0022] Step 600, using the transformation matrix of the robot gripper coordinate system to the robot base coordinate system, the vector is converted into three-dimensional coordinates of the defect in the robot base coordinate system, and then the three-dimensional coordinates of the defect center in the robot base coordinate system are obtained, and the robot gripper can move the defect center to the imaging center of the three-dimensional camera.

[0023] The defect quantification method based on the fusion of three-dimensional cameras and neural networks includes the following steps:

[0024] Step A, the gray-scale image and point cloud data of a larger area collected by the three-dimensional camera are cropped to a smaller area related to the defect, obtaining the gray-scale image and point cloud data of the smaller area;

[0025] Step B, the point cloud data of the cropped smaller area is subjected to surface fitting to obtain a smooth surface model;

[0026] Step C, the instance segmentation neural network model is used to process the gray-scale image of the smaller area to obtain the corresponding mask image;

[0027] Step D, according to the defect edge on the mask image, the sub-point cloud data of the defect edge is extracted from the point cloud data of the smaller area;

[0028] Step E, difference calculation is performed between the sub-point cloud data and the smooth surface model, and the geometric information of the defect is obtained.

[0029] The above-mentioned calculation method for obtaining the defect center coordinates in the two-dimensional image and the action object of the defect quantification method based on the fusion of three-dimensional cameras and neural networks can not be limited to the engine blade and its defects, but also to other test objects Target area, the target area is not necessarily a defect, but also a functional structure deliberately processed, and the defect is not necessarily a pit, but also a crack, a bump and other defects.

[0030] The application will be further described below in conjunction with the drawings and specific embodiments. Additional aspects and advantages of the application will be described in the following description, become apparent from the following description, or be learned through the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The drawings provided in the present application and their related descriptions thereof are used to explain the application, but do not constitute an improper limitation on the application.

[0032] Figure 1 Flow chart of an embodiment of the calculation method for obtaining defect center coordinates from a two-dimensional image according to the present application.

[0033] Figure 2 Flow chart of an embodiment of the defect quantification method based on the fusion of three-dimensional cameras and neural networks according to the present application.

[0034] Figure 3 Flow chart of an embodiment of the image-based automatic engine blade defect detection method according to the present application.

[0035] Figure 4 Perspective view of an embodiment of the engine blade defect detection system according to the present application.

[0036] Figure 5 Top view of an embodiment of the engine blade defect detection system according to the present application.

[0037] Figure 6 Point cloud data image of a larger area obtained by using a three-dimensional camera according to the present application.

[0038] Figure 7 Gray scale image of a smaller area obtained by cutting a gray scale image of a larger area according to the present application.

[0039] Figure 8 Point cloud data image of a smaller area obtained by cutting a point cloud data image of a larger area according to the present application.

[0040] Figure 9 Smooth surface model image obtained by surface fitting on the point cloud data of the smaller area after cutting according to the present application.

[0041] Figure 10 Mask image obtained by using an instance segmentation neural network model to process the gray scale image of the smaller area according to the present application.

[0042] Figure 11 Sub-point cloud data image of the defect edge extracted from the point cloud data image of the smaller area according to the present application.

[0043] The relevant labels in the above figures are as follows: 100 - tray, 200 - robot, 210 - gripper, 220 - mechanical arm, 230 - base, 300 - two-dimensional camera, 400 - three-dimensional camera, 510 - bar light source, 520 - ring light source, 600 - test table, 710 - U-shaped base, 720 - column, 730 - hinged seat, 740 - crossbar. DETAILED DESCRIPTION

[0044] The present application will be described in detail below with reference to the drawings. Those skilled in the art will be able to implement the present application based on these descriptions. Before the present application is described in detail with reference to the drawings, it is particularly important to note that:

[0045] The technical solutions and technical features provided in each part of the present application, including the following description, can be combined with each other without conflict.

[0046] In addition, the embodiments of the present application involved in the following description are generally only a part of the embodiments of the present application, not all. Therefore, all other embodiments obtained by those skilled in the art based on the embodiments in the present application without creative labor should belong to the scope of protection of the present application.

[0047] Regarding the terms and units in the present application. The terms "include", "have" and any variations thereof in the specification and claims of the present application and related parts are intended to cover non-exclusive inclusion.

[0048] Figure 1 The flowchart of an embodiment of the calculation method for obtaining the defect center coordinates from the two-dimensional image of the present application.

[0049] As Figure 1 shown, the embodiment of the calculation method for obtaining the defect center coordinates from the two-dimensional image of the present application is for making the gripper of the robot hold the test object and move its defect center to the imaging center of the three-dimensional camera, and the calculation method comprises steps 100-600, specifically as follows:

[0050] Step 100, multiply the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image with the intrinsic matrix of the two-dimensional camera, and calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system.

[0051] Step 200, according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera, the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system are calculated.

[0052] Step 300: Using the transformation matrix from the 2D camera coordinate system to the robot base coordinate system, the actual 3D coordinates of the defect in the 2D camera coordinate system are transformed into 3D coordinates in the robot base coordinate system. The process of obtaining the transformation matrix from the 2D camera coordinate system to the robot base coordinate system includes steps 310 to 370, as follows:

[0053] Step 310: Use a 2D camera to capture images of the chessboard pattern grasped by the robot gripper in different poses;

[0054] Step 320: Obtain the pixel coordinates of the corner points of the chessboard.

[0055] Use OpenCV's cv2.find Chess board Corners function to extract values ​​from each 2D image I gray,k Extract the corner pixel coordinates P of the chessboard grid. uv,k =(x uv,k ,y uv,k ), where u and v represent the positions of the corner points on the chessboard, and k represents the index of the image;

[0056] Step 330: Establish the mapping relationship between corner pixel coordinates and robot base coordinate system;

[0057] The Zhang Dingyou calibration method is used to establish the mapping relationship between the corner pixel coordinates and the robot base coordinate system. Here, it is assumed that the three-dimensional coordinates of the chessboard grid in the robot base coordinate system are P. uv =(X uv Y uv Z uv ), where Z uv =0;

[0058] Step 340: Solve for the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the 2D camera;

[0059] The intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of a 2D camera are calculated using OpenCV's `calibrate Camera` function, and are expressed as follows:

[0060] H k =K·[R k |T k ];d=[k1,k2,p1,p2,k3];

[0061] In the formula, K is the intrinsic parameter matrix, f x and f y It's the focal length, c x and c y These are the principal point coordinates; H k R is the homography matrix; k Let T be the rotation matrix.k is a 3 x 4 matrix, representing the combination of a rotation matrix and a translation vector, i.e. an extrinsic matrix; d is a distortion coefficient, and k1, k2, k3, k4 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients; k k is a 3 x 4 matrix, representing the combination of a rotation matrix and a translation vector, i.e. an extrinsic matrix; d is a distortion coefficient, and k1, k2, k3, k4 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients;

[0062] Step 350, minimizing the re-projection error between the actual detected corner position and the theoretical corner position, denoted as:

[0063] Minimize∑ k ∑ u,v ||p uv,k -K(R k X uv +T k )|| 2 ;

[0064] Step 360, using the maximum likelihood estimation method to optimize the intrinsic matrix, the distortion coefficient and the extrinsic matrix of the two-dimensional camera;

[0065] Step 370, applying the intrinsic matrix and the extrinsic matrix to the conversion of the corner pixel coordinates, obtaining the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system, denoted as:

[0066] The actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are denoted as:

[0067]

[0068] The three-dimensional coordinates of the defect in the robot base coordinate system are denoted as:

[0069]

[0070] wherein, Z 2D is the distance between the test object and the two-dimensional camera, and (u, v) is the two-dimensional pixel coordinate matrix of the defect.

[0071] Step 400, converting the three-dimensional coordinates of the defect in the robot base coordinate system to the three-dimensional coordinates in the robot gripper coordinate system by using the transformation matrix from the robot base coordinate system to the robot gripper coordinate system;

[0072] The three-dimensional coordinates of the defect in the robot gripper coordinate system are denoted as:

[0073]

[0074] wherein, T base-to-gripper is the transformation matrix from the robot base coordinate system to the robot gripper coordinate system. ​

[0075] Step 500, the three-dimensional coordinates of the defect in the robot gripper coordinate system are subtracted from the origin of the robot gripper coordinate system, and a vector representing the relative position of the defect in the robot gripper coordinate system is calculated;

[0076] The relative position of the defect in the robot gripper coordinate system is a vector is represented as:

[0077]

[0078] In the formula, are unit vectors along the x, y, and z axes, respectively.

[0079] Step 600, using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, the vector is converted into three-dimensional coordinates of the defect in the robot base coordinate system, and then the three-dimensional coordinates of the defect center in the robot base coordinate system are obtained, and the robot gripper can move the defect center to the imaging center of the three-dimensional camera;

[0080] The three-dimensional coordinates of the defect center in the robot base coordinate system (x base , y base , z base ) are represented as:

[0081]

[0082] In the formula, T gripper-to-base The method for obtaining T gripper-to-base is to move the robot gripper to the imaging center of the three-dimensional camera without holding the test object, and obtain the coordinates of the robot gripper through three-dimensional camera imaging, that is, the transformation matrix T camera-to-gripper from the imaging center of the three-dimensional camera to the gripper coordinate system and the robot base coordinate system is established.

[0083] In the above calculation method for obtaining the defect center coordinates from the two-dimensional image, a target detection neural network model is used to detect the defect in the two-dimensional image and output a rectangular coordinate frame of the defect. The two-dimensional pixel coordinate matrix includes the coordinates of the four corner points and the center point of the rectangular coordinate frame. There are two paths to obtain the three-dimensional coordinates of the defect center in the robot base coordinate system: (1) first convert the four corner point coordinates into four three-dimensional coordinates in the robot base coordinate system through T base-to-gripper , T gripper-to-base , T camera-to-gripper , and then calculate the three-dimensional coordinates of the defect center from the four three-dimensional coordinates; (2) directly convert the center point coordinates to obtain the three-dimensional coordinates of the defect center through T base-to-gripper , T gripper-to-base It is found through verification that the second path has smaller error and higher efficiency.

[0084] Figure 2 Flow chart of an embodiment of the defect quantification method based on fusion of three-dimensional camera and neural network of the present application.

[0085] As shown in Figure 2 , an embodiment of the defect quantification method based on fusion of three-dimensional camera and neural network of the present application comprises steps A-E, as follows:

[0086] Step A, according to the imaging center of the three-dimensional camera and the defect size calculated from the two-dimensional image, the gray-scale image and the point cloud data of a larger area collected by the three-dimensional camera are cropped to a smaller area related to the defect, to obtain the gray-scale image and the point cloud data of the smaller area; the three-dimensional camera adopts a monocular structured light camera;

[0087] The defect size is calculated from the four three-dimensional coordinates of the rectangular coordinate frame of the defect output by the target detection neural network model processing the two-dimensional image in the robot base coordinate system, as follows:

[0088] Let the four three-dimensional coordinates of the four corner points in the robot base coordinate system be and

[0089] Calculate the maximum and minimum values of the four three-dimensional coordinates on the x-axis:

[0090]

[0091] Calculate the maximum and minimum values of the four three-dimensional coordinates on the y-axis:

[0092]

[0093] Calculate the maximum and minimum values of the four three-dimensional coordinates on the z-axis:

[0094]

[0095] According to the maximum and minimum values of the coordinate axes, calculate the length, width and height of the defect:

[0096] Length: length = max x -min x ;

[0097] Width: width = max y -min y ;

[0098] Height: height = max z -min z ;

[0099] The four three-dimensional coordinates also have two paths, which are: (1) sequentially passing the four corner point coordinates through T camera-to-gripper , T base-to-gripper , T gripper-to-base to obtain; (2) passing the four corner point coordinates through T camera-to-gripper to obtain. It is verified that the errors of the two paths are almost the same, but the efficiency of the second path is obviously higher.

[0100] Step B, surface fitting is performed on the point cloud data of the cut smaller area to obtain a smooth surface model; specifically comprising the following steps B1-B4:

[0101] Step B1, effective point cloud data is screened out, and the expression is calculated as: P valid =Mask·P raw ; in the formula, P valid is the screened effective point cloud data; P raw is the point cloud data of the smaller area; Mask is a mask obtained from the point cloud data collected by the three-dimensional camera;

[0102] Step B2, surface fitting is performed on the screened effective point cloud data to obtain a reference surface;

[0103] Step B3, the BFGS algorithm in the minimization method is used to minimize the target function of the fitting error to obtain the optimal fitting parameter: wherein the expression of the target function f(c) of the fitting error is:

[0104]

[0105] In the formula, c=[A, B, C, D, E, F] is the fitting parameter; (x n , y n , z n ) is the coordinate of the nth point cloud data.

[0106] Step B4, the depth of the fitting surface is calculated using the optimal fitting parameter to obtain a smooth surface model;

[0107] The calculation expression of the depth z of the fitting surface using the optimal fitting parameter is:

[0108] fit z =A * ·x 2 +B * ·y 2 +C * ·x·y+D * ·x+E * ·y+F * ;

[0109] In the formula, c* = [A*, B*, C*, D*, E*, F*] is the optimal fitting parameter; (x, y, z) is the coordinate of the depth of the fitting surface.

[0110] Step C, processing the gray image of the smaller area using the instance segmentation neural network model to obtain a corresponding mask image.

[0111] Step D, according to the defect edge on the mask image, the sub-point cloud data of the defect edge is extracted from the point cloud data of the smaller area; according to the sub-point cloud data, the perimeter, diameter and volume of the defect can be calculated.

[0112] Step E, difference calculation is performed between the sub-point cloud data and the smooth surface model to obtain the depth deviation of the defect, and then the actual depth of the defect is determined, and finally the geometric information of the defect is obtained.

[0113] The calculation expression of the depth deviation △D of the defect is: △D = P valid -fit z .

[0114] Figure 3 The flowchart of the embodiment of the image-based engine blade defect automatic detection method of the application.

[0115] As Figure 3 shown, the embodiment of the image-based engine blade defect automatic detection method of the application includes moving the blade to the detection site by using the robot gripper and taking a photo of the blade at the detection site by using a two-dimensional camera and a three-dimensional camera, and further includes steps (1) to (5), which are specifically as follows:

[0116] Step (1), taking a photo of the blade gripped by the robot gripper by using the two-dimensional camera to obtain a plurality of two-dimensional images covering the surface of the blade.

[0117] Preferably, the surface of the blade is divided into a plurality of detection points, and the gripper moves these detection points to the imaging center of the two-dimensional camera for photographing. The robot gripper grips the same blade twice, and the positions of the two grips are different, so that the surface covered by the gripper during the first grip can also be detected, so that the obtained two-dimensional image covers all key areas and avoids blind areas.

[0118] Preferably, the plurality of two-dimensional images includes furnace batch number images and local position images; wherein, for the furnace batch number images, the recognition of the furnace batch number is carried out using an OCR algorithm processing; for the local position images, the defect detection is carried out using a target detection neural network model, whereby the recognized furnace batch number will be bound with subsequent detection data (including defect detection results, quantitative data, etc.), ensuring that all detection information can be traced back to a specific production batch. Wherein, the OCR algorithm adopts DBM (Deep Bidirectional Model), uses bidirectional modeling technology for feature extraction and sequence modeling, captures the context relationship between characters, can deep feature learning to improve recognition accuracy and robustness, has strong generalization ability to adapt to various fonts and backgrounds, uses efficient training and reasoning mechanism to ensure fast processing of image data, so as to realize high-precision recognition under complex conditions.

[0119] Step (2), using a target detection neural network model to detect defects in the plurality of two-dimensional images.

[0120] The architecture of the target detection neural network model includes a backbone network, a neck network and a detection head, the backbone network is responsible for extracting image features, the neck network further processes and fuses these features, and the detection head generates the final detection results. The target detection neural network model adopts a continuous double assignment strategy without NMS (non-maximum suppression) training, which significantly improves the detection accuracy of small size and overlapping defects, can accurately identify small size defects, and effectively handles overlapping target problems, enhances the adaptability to complex background and high density targets. In addition, the NMS-free strategy reduces the possibility of false positives and false negatives, ensures the accuracy and stability of the detection results, making it more suitable for efficient and accurate detection of blade defects.

[0121] Step (3), using the above-mentioned calculation method of obtaining the center coordinates of the defects from the two-dimensional images, for the two-dimensional images detected with defects, calculating the three-dimensional coordinates of the defect center in the robot base coordinate system, and then moving the defect center to the imaging center of the three-dimensional camera by the gripper.

[0122] Step (4), acquiring the gray-scale image and point cloud data of the defect by the three-dimensional camera.

[0123] Step (5), using the above-mentioned defect quantification method based on the fusion of three-dimensional camera and neural network to process the gray-scale image and point cloud data of the defect, obtaining the geometric information of the defect.

[0124] Figure 4 A perspective view of an embodiment of the engine blade defect detection system of the present application. Figure 5 A top view of an embodiment of the engine blade defect detection system of the present application.

[0125] As shown in Figures 4-5 the engine blade defect detection system includes a tray 100, a robot 200, a two-dimensional camera 300, a three-dimensional camera 400, a light supplement source, a terminal device and a test bench 600. The surface of the test bench 600 is assembled by T-shaped grooves. The tray 100 is used to store blades, and the tray 100 is provided with blade storage grooves.

[0126] The robot 200 is used to clamp and move the blades. The robot 200 has a clamping jaw 210, a mechanical arm 220 and a base 230. The mechanical arm 220 can drive the blades to rotate and lift. The base 230 is detachably connected with the T-shaped grooves and can slide horizontally along the T-shaped grooves.

[0127] The two-dimensional camera 300 and the three-dimensional camera 400 are used to take pictures of the blades clamped by the clamping jaw 210. The three-dimensional camera 400 is a monocular structured light camera. The two-dimensional camera 300 is arranged in front of the robot 200, the tray 100 is arranged beside the robot 200, and the three-dimensional camera 400 is arranged between the tray 100 and the two-dimensional camera 300 and is arranged obliquely towards the robot 200.

[0128] The light supplement source is used to supplement light when the two-dimensional camera 300 and the three-dimensional camera 400 take pictures. The light supplement source includes a strip light source 510 and a ring light source 520. The ring light source 520 is arranged in front of the two-dimensional camera 300, and the strip light source 510 is arranged between the two-dimensional camera 300 and the robot 200. The strip light source 510 is at least two.

[0129] The two-dimensional camera 300, the three-dimensional camera 400 and the light supplement source are fixed on the test bench 600 through adjustable supports. The adjustable support includes a U-shaped base 710, a stand 720 and a hinged seat 730. The U-shaped base 710 is detachably connected with the T-shaped grooves and can slide horizontally along the T-shaped grooves. The two-dimensional camera 300, the three-dimensional camera 400 and the light supplement source are detachably connected with the stand 720 through the vertical through holes of the hinged seat 730 and can slide vertically and rotate horizontally along the stand 720. Further, the hinged seat 730 connected with the two-dimensional camera 300 or the three-dimensional camera 400 also includes a horizontal through hole. The adjustable support also includes a crossbar 740, which is detachably connected with the stand 720 through the horizontal through hole of the hinged seat 730 and can move along the horizontal through hole. The end of the crossbar 740 is connected with the two-dimensional camera 300 or the three-dimensional camera 400.

[0130] The terminal device communicates with the two-dimensional camera 300, the three-dimensional camera 400 and the robot 200 through a wireless network and controls the clamping and movement of the blades by the clamping jaw 210.

[0131] The beneficial effects of the present application are illustrated below by specific test data. The preferred technical solution with small error and higher efficiency is adopted, i.e. the three-dimensional coordinates of the defect center are directly obtained from the center point coordinates of the rectangular coordinate frame by T camera-to-gripper , T base-to-gripper , T gripper-to-base transformation, and the three-dimensional coordinates for calculating the defect size are obtained from the four corner point coordinates of the rectangular coordinate frame by T camera-to-gripper transformation. The specific calculation process and data are shown as follows:

[0132] 1. The parameters of the two-dimensional camera are calculated as follows:

[0133] Intrinsic matrix

[0134] Distortion coefficient d = [0.20334415671740425, 4.904385112163188, -0.0022254112636877267, 1.4572813224837689e-05, -1037.0344399454984];

[0135] Reprojection error = 0.047132498496030244;

[0136]

[0137] 2. The actual three-dimensional camera coordinates of the two-dimensional pixel coordinate matrix in the two-dimensional camera coordinate system are calculated as follows:

[0138] The target detection neural network model is used to detect the defects in the two-dimensional image and output the rectangular coordinate frame of the defects, and the four corner point coordinates of the rectangular coordinate frame are [1136.0, 1818.0], [1186.0, 1818.0], [1136.0, 1872.0], [1186.0, 1872.0], and the center point coordinates are [1161.0, 1845.0];

[0139] The actual three-dimensional camera coordinates of the four corner point coordinates in the two-dimensional camera coordinate system are calculated as follows:

[0140] The three-dimensional coordinates corresponding to [1136.0, 1818.0] are [-14.37580796, -5.92899294, 270.5];

[0141] The three-dimensional coordinates corresponding to [1186.0, 1818.0] are [-13.83966873, -5.92899294, 270.5];

[0142] [1136.0, 1872.0] corresponding three-dimensional coordinates: [-14.37580796, -5.34946993, 270.5];

[0143] [1186.0, 1872.0] corresponding three-dimensional coordinates: [-13.83966873, -5.34946993, 270.5];

[0144] The actual three-dimensional camera coordinates of the center point coordinates in the two-dimensional camera coordinate system are calculated as follows:

[0145] [-14.10773834, -5.63923143, 270.5].

[0146] 3. The three-dimensional coordinates of the center point coordinates under the robot base coordinate system are calculated as follows:

[0147] Through T camera-to-base The center point coordinates are converted into three-dimensional coordinates under the robot base coordinate system: [-631.83898003,

[0148] -128.41387102, 254.0063209].

[0149] 4. The three-dimensional coordinates of the center point coordinates under the robot gripper coordinate system are calculated as follows:

[0150]

[0151] Through T base-to-gripper The three-dimensional coordinates of the center point coordinates under the robot base coordinate system are converted into three-dimensional coordinates under the robot gripper coordinate system: [-3.98719122, 37.2480781, 36.18581824].

[0152] 5. The three-dimensional coordinates of the center point coordinates under the robot base coordinate system are calculated as follows:

[0153]

[0154] Through T gripper-to-base The three-dimensional coordinates of the center point coordinates under the robot gripper coordinate system are converted into three-dimensional coordinates under the robot base coordinate system: [-651.80876029, 255.67971206, 319.5136791].

[0155] 6. The gripper holds the blade to [-651.80876029, 255.67971206, 319.5136791], and then a three-dimensional camera is used to collect a large area of gray scale image and point cloud data image (as shown in Figure 6 ).

[0156] 7. Calculate the defect size as follows:

[0157] By T camera-to-base Convert the four corner point coordinates into three-dimensional coordinates in the robot base coordinate system, respectively:

[0158] [1136.0, 1818.0] corresponding three-dimensional coordinates: [-631.83906977, -128.14745721, 254.29644636];

[0159] [1186.0, 1818.0] corresponding three-dimensional coordinates: [-631.84835501, -128.68251816, 254.2949805];

[0160] [1136.0, 1872.0] corresponding three-dimensional coordinates: [-631.82960527, -128.14523899, 253.71765873];

[0161] [1186.0, 1872.0] corresponding three-dimensional coordinates: [-631.83889034, -128.68031196, 253.71616966];

[0162] Due to slight errors in the conversion data, the three-dimensional coordinates of the four corner points after compensation and lengthening and widening by 1mm are:

[0163] [1136.0, 1818.0] corresponding three-dimensional coordinates: [-631.83906977, -127.14745721, 255.29644636];

[0164] [1186.0, 1818.0] corresponding three-dimensional coordinates: [-631.84835501, -129.68251816, 255.2949805];

[0165] [1136.0, 1872.0] corresponding three-dimensional coordinates: [-631.82960527, -127.14523899, 252.71765873];

[0166] [1186.0, 1872.0] corresponding three-dimensional coordinates: [-631.83889034, -129.68031196, 252.71616966];

[0167] The length, width and height of the defect are calculated as follows: length ≈ 0.0187mm, width ≈ 2.5372mm, height ≈ 2.5802mm.

[0168] 8. According to the length, width and height of the defect, the gray-scale image and the point cloud data image of the larger area are cut to obtain the gray-scale image and the point cloud data image of the smaller area, respectively as shown in Figure 7 and Figure 8 .

[0169] 9. The point cloud data of the cut smaller area is subjected to surface fitting to obtain a smooth surface model as shown in Figure 9 .

[0170] 10. The gray-scale image of the smaller area is processed using an instance segmentation neural network model to obtain a mask map as shown in Figure 10 .

[0171] 11. According to the defect edge on the mask map, the sub-point cloud data image of the defect edge is extracted from the point cloud data image of the smaller area as shown in Figure 11 .

[0172] 12. The sub-point cloud data is subjected to difference calculation with the smooth surface model, and finally the type of the defect is obtained as a pit with a pit diameter of 0.549 mm and a pit depth of -0.158 mm, and the judgment result is unqualified.

[0173] The above describes the relevant content of the present application. Those skilled in the art will be able to implement the present application based on these descriptions. Based on the above content of the present application, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

Claims

1. A defect quantification method based on the fusion of 3D camera and neural network, characterized in that: Includes the following steps: Step A: The grayscale image and point cloud data of the larger area acquired by the 3D camera are cropped into a smaller area related to the defect, resulting in the grayscale image and point cloud data of the smaller area. Step B involves performing surface fitting on the point cloud data of the cropped smaller region to obtain a smooth surface model. Step C: Use an instance segmentation neural network model to process the grayscale image of a smaller region to obtain the corresponding mask image; Step D: Based on the defect edges on the mask image, extract the sub-point cloud data of the defect edges from the point cloud data of a smaller area; Step E involves calculating the difference between the sub-point cloud data and the smooth surface model to obtain the geometric information of the defect. The process involves calculating the three-dimensional coordinates of the defect center in the robot's base coordinate system from the two-dimensional image, then moving the defect center to the imaging center of the three-dimensional camera using the gripper; and finally cropping based on the imaging center of the three-dimensional camera and the defect size calculated from the two-dimensional image. The method for calculating the three-dimensional coordinates of the defect center in the robot base coordinate system includes the following steps: Step 100: Multiply the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image with the intrinsic parameter matrix of the two-dimensional camera to calculate the normalized planar coordinates of the defect in the two-dimensional camera coordinate system. Step 200: Calculate the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system based on the normalized planar coordinates and the distance between the test object and the two-dimensional camera. Step 300: Using the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system, the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are converted into three-dimensional coordinates in the robot base coordinate system. Step 400: Using the transformation matrix from the robot base coordinate system to the robot gripper coordinate system, the three-dimensional coordinates of the defect in the robot base coordinate system are transformed into three-dimensional coordinates in the robot gripper coordinate system. Step 500: Subtract the three-dimensional coordinates of the defect in the robot gripper coordinate system from the origin of the robot gripper coordinate system to calculate the vector representing the relative position of the defect in the robot gripper coordinate system. Step 600: Using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, the vector is converted into the three-dimensional coordinates of the defect in the robot base coordinate system, thereby obtaining the three-dimensional coordinates of the defect center in the robot base coordinate system.

2. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 1, characterized in that: The 3D camera is a monocular structured light camera.

3. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 2, characterized in that: The size of the defect is calculated from the four three-dimensional coordinates of the four corner points of the rectangular frame of the defect output by the target detection neural network model in the robot base coordinate system.

4. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 1, characterized in that: Step B includes the following steps: Filter out the valid point cloud data; A baseline surface is obtained by performing surface fitting on the filtered valid point cloud data. The BFGS algorithm from the minimization method is used to minimize the objective function of the fitting error to obtain the optimal fitting parameters: The depth of the fitted surface is calculated using the optimal fitting parameters, thus obtaining the smooth surface model.

5. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 4, characterized in that: The calculation expression for selecting valid point cloud data is: P valid =Mask·P raw ; In the formula, P valid It is the filtered, valid point cloud data; P raw It is point cloud data of a smaller area; Mask is a mask obtained from point cloud data collected by a 3D camera.

6. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 5, characterized in that: The objective function f(c) for the fitting error is expressed as: In the formula, c = [A, B, C, D, E, F] are the fitting parameters; (x n y n , z n ) represents the coordinates of the nth point cloud data.

7. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 6, characterized in that: The expression for calculating the depth z of the fitted surface using the optimal fitting parameters is as follows: fit z =A * ·x 2 +B * ·y 2 +C * ·x·y+D * ·x+E * ·y+F * ; In the formula, c*=[A*,B*,C*,D*,E*,F*] are the optimal fitting parameters; (x, y, z) are the coordinates at the depth of the fitted surface.

8. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 7, characterized in that: In step D, the perimeter, diameter, and volume of the defect are calculated based on the sub-point cloud data; in step E, the difference between the sub-point cloud data and the smooth surface model is calculated to obtain the depth deviation of the defect, and then the actual depth of the defect is determined.

9. The defect quantification method based on the fusion of a 3D camera and a neural network as described in claim 8, characterized in that: The expression for calculating the depth deviation ΔD of the defect in step E is: ΔD = P valid -fit z .

10. A method for detecting defects in engine blades, characterized in that: The defect quantification method described in any one of claims 1-9 is used to process the grayscale image and point cloud data of the engine blade to obtain the geometric information of the defect.

Images