Method for obtaining defect center coordinates from two-dimensional images and applications thereof

Abstract

The application belongs to the technical field of image processing, and discloses a calculation method for obtaining a defect center coordinate from a two-dimensional image and application thereof, including the following steps: step 100, calculating to obtain normalized plane coordinates of the defect in a two-dimensional camera coordinate system; step 200, calculating actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system; step 300, converting the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system into three-dimensional coordinates under a robot base coordinate system; step 400, converting the three-dimensional coordinates of the defect under the robot base coordinate system into three-dimensional coordinates under a robot gripper coordinate system; step 500, subtracting the three-dimensional coordinates of the defect under the robot gripper coordinate system from the origin of the robot gripper coordinate system to calculate a vector representing the relative position of the defect under the robot gripper coordinate system; and step 600, converting the vector into three-dimensional coordinates of the defect under the robot base coordinate system to obtain three-dimensional coordinates of the defect center under the robot base coordinate system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a calculation method for obtaining a defect center coordinate from a two-dimensional image and application thereof. BACKGROUND

[0002] As the key power device of an aircraft, aerospace engine is known as the "heart" of the aircraft and rocket. Its independent research and development is of great significance to the national defense construction, energy security and sustainable development of the environment. Among them, the blade is the core component of the aerospace engine for guiding airflow and generating power. The detection of its key quality characteristics directly affects the service safety and reliability of the aircraft.

[0003] Among the many components of the engine, the number of blades is large, and its manufacturing quantity accounts for about one-third of the entire machine manufacturing quantity. The blade has the characteristics of large quantity, complex shape, high precision requirement, difficult processing and frequent failure, and is a key component of the engine. The engine relies on numerous blades to complete the compression and expansion of gas to generate powerful power to propel the aircraft forward. Therefore, the quality of the blade is a key factor affecting the energy conversion efficiency and working performance of the engine.

[0004] At present, the detection means for blade surface defects mainly relies on manual visual inspection. From the perspective of detection quality, the detection result is greatly affected by human factors. From the perspective of detection efficiency, the blade appearance is complex, and the manual detection efficiency is low. From the perspective of detection data, the manual detection result is only a qualitative judgment, and lacks quantitative data. The result is difficult to be stored digitally, which is not conducive to the later quality problem tracing. From the perspective of detection quantity, there is still a lack of intelligent detection capability for batch blade surface defects on site at present. The manual visual inspection method has reached the production capacity bottleneck and cannot meet the needs of production and delivery. SUMMARY

[0005] The main purpose of the present application is to provide an image-based engine blade defect automatic detection method and an engine blade defect detection system. The blade surface defects are detected digitally, automatically and intelligently to realize stable quality, high efficiency, high traceability and quantitative detection results.

[0006] In order to achieve the above purpose, the technical scheme of the image-based engine blade defect automatic detection method and the engine blade defect detection system provided by the present application is as follows:

[0007] The image-based engine blade defect automatic detection method comprises the steps of moving the blade to the detection site by using the robot gripper and taking a photo of the blade at the detection site by using a two-dimensional camera and a three-dimensional camera. The method further comprises the following steps:

[0008] Step (1), a two-dimensional camera is used to take pictures of the blade held by the robot gripper to obtain a plurality of two-dimensional images covering the surface of the blade;

[0009] Step (2), a target detection neural network model is used to detect defects in the plurality of two-dimensional images;

[0010] Step (3), for the two-dimensional images in which defects are detected, three-dimensional coordinates of the defect center in the robot base coordinate system are calculated, and then the gripper is caused to move the defect center to the imaging center of the three-dimensional camera;

[0011] Step (4), a three-dimensional camera is used to collect a grayscale image and point cloud data of the defect;

[0012] Step (5), a defect quantification method based on the fusion of a three-dimensional camera and a neural network is used to process the grayscale image and point cloud data of the defect to obtain geometric information of the defect.

[0013] The engine blade defect detection system comprises a tray for storing blades, wherein a blade storage groove is arranged on the tray; a robot for clamping and moving the blades, wherein the robot has a gripper and a mechanical arm; a two-dimensional camera for taking pictures of the blades held by the gripper; a three-dimensional camera for taking pictures of the blades held by the gripper; and a light supplementing light source for supplementing light when the two-dimensional camera and the three-dimensional camera take pictures.

[0014] In the above image-based engine blade defect automatic detection method, when steps (3) and (5) adopt the following calculation method for obtaining the coordinates of the defect center in the two-dimensional image and the defect quantification method based on the fusion of a three-dimensional camera and a neural network, the calculation method is simple, and the calculation efficiency and accuracy are high.

[0015] Therefore, the present application provides a calculation method for obtaining the coordinates of the defect center in the two-dimensional image and a defect quantification method based on the fusion of a three-dimensional camera and a neural network, and the technical solutions are as follows:

[0016] The calculation method for obtaining the coordinates of the defect center in the two-dimensional image is used to make the gripper of the robot hold a test object and move the defect center of the test object to the imaging center of the three-dimensional camera, and the calculation method comprises the following steps:

[0017] Step 100, multiplying the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image and the intrinsic matrix of the two-dimensional camera to calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system;

[0018] Step 200, calculating the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera;

[0019] Step 300, using the transformation matrix of the two-dimensional camera coordinate system to the robot base coordinate system, the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are converted into three-dimensional coordinates in the robot base coordinate system;

[0020] Step 400, using the transformation matrix of the robot base coordinate system to the robot gripper coordinate system, the three-dimensional coordinates of the defect in the robot base coordinate system are converted into three-dimensional coordinates in the robot gripper coordinate system;

[0021] Step 500, the three-dimensional coordinates of the defect in the robot gripper coordinate system are subtracted from the origin of the robot gripper coordinate system, and a vector representing the relative position of the defect in the robot gripper coordinate system is calculated;

[0022] Step 600, using the transformation matrix of the robot gripper coordinate system to the robot base coordinate system, the vector is converted into three-dimensional coordinates of the defect in the robot base coordinate system, and then the three-dimensional coordinates of the defect center in the robot base coordinate system are obtained, and the robot gripper can move the defect center to the imaging center of the three-dimensional camera.

[0023] The defect quantification method based on the fusion of three-dimensional cameras and neural networks includes the following steps:

[0024] Step A, the gray-scale image and point cloud data of a larger area collected by the three-dimensional camera are cropped to a smaller area related to the defect, obtaining the gray-scale image and point cloud data of the smaller area;

[0025] Step B, the point cloud data of the cropped smaller area is subjected to surface fitting to obtain a smooth surface model;

[0026] Step C, the instance segmentation neural network model is used to process the gray-scale image of the smaller area to obtain the corresponding mask image;

[0027] Step D, according to the defect edge on the mask image, the sub-point cloud data of the defect edge is extracted from the point cloud data of the smaller area;

[0028] Step E, difference calculation is performed between the sub-point cloud data and the smooth surface model, and the geometric information of the defect is obtained.

[0029] The above-mentioned calculation method for obtaining the defect center coordinates in the two-dimensional image and the action object of the defect quantification method based on the fusion of three-dimensional cameras and neural networks can not be limited to the engine blade and its defects, but also to other test objects Target area, the target area is not necessarily a defect, but also a functional structure deliberately processed, and the defect is not necessarily a pit, but also a crack, a bump and other defects.

[0030] The application will be further described below in conjunction with the drawings and specific embodiments. Additional aspects and advantages of the application will be described in the following description, become apparent from the following description, or be learned through the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The drawings provided in the present application and their related descriptions thereof are used to explain the application, but do not constitute an improper limitation on the application.

[0032] Figure 1 Flow chart of an embodiment of the calculation method for obtaining defect center coordinates from a two-dimensional image according to the present application.

[0033] Figure 2 Flow chart of an embodiment of the defect quantification method based on the fusion of three-dimensional cameras and neural networks according to the present application.

[0034] Figure 3 Flow chart of an embodiment of the image-based automatic engine blade defect detection method according to the present application.

[0035] Figure 4 Perspective view of an embodiment of the engine blade defect detection system according to the present application.

[0036] Figure 5 Top view of an embodiment of the engine blade defect detection system according to the present application.

[0037] Figure 6 Point cloud data image of a larger area obtained by using a three-dimensional camera according to the present application.

[0038] Figure 7 Gray scale image of a smaller area obtained by cutting a gray scale image of a larger area according to the present application.

[0039] Figure 8 Point cloud data image of a smaller area obtained by cutting a point cloud data image of a larger area according to the present application.

[0040] Figure 9 Smooth surface model image obtained by surface fitting on the point cloud data of the smaller area after cutting according to the present application.

[0041] Figure 10 Mask image obtained by using an instance segmentation neural network model to process the gray scale image of the smaller area according to the present application.

[0042] Figure 11 Sub-point cloud data image of the defect edge extracted from the point cloud data image of the smaller area according to the present application.

[0043] The relevant labels in the above figures are as follows: 100 - tray, 200 - robot, 210 - gripper, 220 - mechanical arm, 230 - base, 300 - two-dimensional camera, 400 - three-dimensional camera, 510 - bar light source, 520 - ring light source, 600 - test table, 710 - U-shaped base, 720 - column, 730 - hinged seat, 740 - crossbar. DETAILED DESCRIPTION

[0044] The present application will be described in detail below with reference to the drawings. Those skilled in the art will be able to implement the present application based on these descriptions. Before the present application is described in detail with reference to the drawings, it is important to specifically point out that:

[0045] The technical solutions and technical features provided in each part of the present application, including the following description, can be combined with each other without conflict.

[0046] In addition, the embodiments of the present application involved in the following description are generally only a part of the embodiments of the present application, not all. Therefore, all other embodiments obtained by those skilled in the art based on the embodiments in the present application without creative labor should belong to the scope of protection of the present application.

[0047] Regarding the terms and units in the present application. The terms "include", "have" and any variations thereof in the specification and claims of the present application and related parts are intended to cover non-exclusive inclusion.

[0048] Figure 1 The flowchart of an embodiment of the calculation method for obtaining the defect center coordinates from the two-dimensional image of the present application.

[0049] As Figure 1 shown, the embodiment of the calculation method for obtaining the defect center coordinates from the two-dimensional image of the present application is for making the gripper of the robot hold the test object and move its defect center to the imaging center of the three-dimensional camera, and the calculation method comprises steps 100-600, specifically as follows:

[0050] Step 100, multiply the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image with the intrinsic matrix of the two-dimensional camera, and calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system.

[0051] Step 200, according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera, calculate the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system.

[0052] Step 300: Using the transformation matrix from the 2D camera coordinate system to the robot base coordinate system, the actual 3D coordinates of the defect in the 2D camera coordinate system are transformed into 3D coordinates in the robot base coordinate system. The process of obtaining the transformation matrix from the 2D camera coordinate system to the robot base coordinate system includes steps 310 to 370, as follows:

[0053] Step 310: Use a 2D camera to capture images of the chessboard pattern grasped by the robot gripper in different poses;

[0054] Step 320: Obtain the pixel coordinates of the corner points of the chessboard.

[0055] Use OpenCV's cv2.find Chess board Corners function to extract values ​​from each 2D image I gray,k Extract the corner pixel coordinates P of the chessboard grid. uv,k =(x uv,k ,y uv,k ), where u and v represent the positions of the corner points on the chessboard, and k represents the index of the image;

[0056] Step 330: Establish the mapping relationship between corner pixel coordinates and robot base coordinate system;

[0057] The Zhang Dingyou calibration method is used to establish the mapping relationship between the corner pixel coordinates and the robot base coordinate system. Here, it is assumed that the three-dimensional coordinates of the chessboard grid in the robot base coordinate system are P. uv =(X uv Y uv Z uv ), where Z uv =0;

[0058] Step 340: Solve for the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the 2D camera;

[0059] The intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of a 2D camera are calculated using OpenCV's `calibrate Camera` function, and are expressed as follows:

[0060] H k =K·[R k |T k ];d=[k1,k2,p1,p2,k3];

[0061] In the formula, K is the intrinsic parameter matrix, f x and f y It's the focal length, c x and c y These are the principal point coordinates; H k R is the homography matrix; k Let T be the rotation matrix.k is a 3x4 matrix, representing the combination of rotation matrix and translation vector, i.e. the extrinsic matrix; d is the distortion coefficient, k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients; k k is a 3x4 matrix, representing the combination of rotation matrix and translation vector, i.e. the extrinsic matrix; d is the distortion coefficient, k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients;

[0062] Step 350, minimizing the re-projection error between the actual detected corner position and the theoretical corner position, denoted as:

[0063] Minimize∑ k ∑ u, v||P uv,k -K(R k X uv +T k )|| 2 ;

[0064] Step 360, using the maximum likelihood estimation method to optimize the intrinsic matrix, distortion coefficient and extrinsic matrix of the two-dimensional camera;

[0065] Step 370, applying the intrinsic matrix and extrinsic matrix to the conversion of the corner pixel coordinates, obtaining the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system, denoted as:

[0066] The actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are denoted as:

[0067]

[0068] The three-dimensional coordinates of the defect in the robot base coordinate system are denoted as:

[0069]

[0070] wherein, Z 2D is the distance between the test object and the two-dimensional camera, (u, v) is the two-dimensional pixel coordinate matrix of the defect.

[0071] Step 400, using the transformation matrix from the robot base coordinate system to the robot gripper coordinate system to convert the three-dimensional coordinates of the defect in the robot base coordinate system to the three-dimensional coordinates in the robot gripper coordinate system;

[0072] The three-dimensional coordinates of the defect in the robot gripper coordinate system are denoted as:

[0073]

[0074] wherein, T base-to-gripper is the transformation matrix from the robot base coordinate system to the robot gripper coordinate system. ​

[0075] Step 500, the three-dimensional coordinates of the defect in the robot gripper coordinate system are subtracted from the origin of the robot gripper coordinate system, and a vector representing the relative position of the defect in the robot gripper coordinate system is calculated;

[0076] The relative position of the defect in the robot gripper coordinate system is a vector is expressed as:

[0077]

[0078] In the formula, are unit vectors along the x, y, and z axes, respectively.

[0079] Step 600, using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, the vector is converted into three-dimensional coordinates of the defect in the robot base coordinate system, and then the three-dimensional coordinates of the defect center in the robot base coordinate system are obtained, and the robot gripper can move the defect center to the imaging center of the three-dimensional camera;

[0080] The three-dimensional coordinates of the defect center in the robot base coordinate system (x base , y base , z base ) are expressed as:

[0081]

[0082] In the formula, T gripper-to-base The method for obtaining T gripper-to-base is to move the robot gripper to the imaging center of the three-dimensional camera without holding the test object, and obtain the coordinates of the robot gripper through three-dimensional camera imaging, that is, the transformation matrix T camera-to-gripper from the imaging center of the three-dimensional camera to the gripper coordinate system and the robot base coordinate system is established.

[0083] In the above calculation method for obtaining the defect center coordinates from the two-dimensional image, a target detection neural network model is used to detect the defect in the two-dimensional image and output a rectangular coordinate frame of the defect. The two-dimensional pixel coordinate matrix includes the coordinates of the four corner points and the center point of the rectangular coordinate frame. There are two paths to obtain the three-dimensional coordinates of the defect center in the robot base coordinate system: (1) first convert the four corner point coordinates into four three-dimensional coordinates in the robot base coordinate system through T base-to-gripper , T gripper-to-base , T camera-to-gripper , and then calculate the three-dimensional coordinates of the defect center from the four three-dimensional coordinates; (2) directly convert the center point coordinates through T base-to-gripper , T gripper-to-base to obtain the three-dimensional coordinates of the defect center. It is found through verification that the second path has smaller error and higher efficiency.

[0084] Figure 2 Flow chart of an embodiment of the defect quantification method based on fusion of three-dimensional camera and neural network of the present application.

[0085] As shown in Figure 2 , an embodiment of the defect quantification method based on fusion of three-dimensional camera and neural network of the present application comprises steps A-E, as follows:

[0086] Step A, according to the imaging center of the three-dimensional camera and the defect size calculated from the two-dimensional image, the gray-scale image and the point cloud data of a larger area collected by the three-dimensional camera are cropped to a smaller area related to the defect, to obtain the gray-scale image and the point cloud data of the smaller area; the three-dimensional camera adopts a monocular structured light camera;

[0087] The defect size is calculated from the four three-dimensional coordinates of the rectangular coordinate frame of the defect output by the target detection neural network model processing the two-dimensional image in the robot base coordinate system, as follows:

[0088] Let the four three-dimensional coordinates of the four corner points in the robot base coordinate system be and

[0089] Calculate the maximum and minimum values of the four three-dimensional coordinates on the x-axis:

[0090]

[0091] Calculate the maximum and minimum values of the four three-dimensional coordinates on the y-axis:

[0092]

[0093] Calculate the maximum and minimum values of the four three-dimensional coordinates on the z-axis:

[0094]

[0095] According to the maximum and minimum values of the coordinate axes, calculate the length, width and height of the defect:

[0096] Length: length = max x -min x ;

[0097] Width: width = max y -min y ;

[0098] Height: height = max z -min z ;

[0099] The four three-dimensional coordinates also have two paths, which are: (1) sequentially passing the four corner point coordinates through T camera-to-gripper , T base-to-gripper , T gripper-to-base to obtain; (2) passing the four corner point coordinates through T camera-to-gripper to obtain. It is verified that the errors of the two paths are almost the same, but the efficiency of the second path is obviously higher.

[0100] Step B, surface fitting is performed on the point cloud data of the cut smaller area to obtain a smooth surface model; specifically comprising the following steps B1-B4:

[0101] Step B1, effective point cloud data is screened out, and the expression is calculated as: P valid =Mask·P raw ; in the formula, P valid is the screened effective point cloud data; P raw is the point cloud data of the smaller area; Mask is a mask obtained from the point cloud data collected by the three-dimensional camera;

[0102] Step B2, surface fitting is performed on the screened effective point cloud data to obtain a reference surface;

[0103] Step B3, the BFGS algorithm in the minimization method is used to minimize the target function of the fitting error to obtain the optimal fitting parameter: wherein the expression of the target function f(c) of the fitting error is:

[0104]

[0105] In the formula, c=[A, B, C, D, E, F] is the fitting parameter; (x n , y n , z n ) is the coordinate of the nth point cloud data.

[0106] Step B4, the depth of the fitting surface is calculated using the optimal fitting parameter to obtain a smooth surface model;

[0107] The calculation expression of the depth z of the fitting surface using the optimal fitting parameter is:

[0108] fit z =A * ·x 2 +B * ·y 2 +C * ·x·y+D * ·x+E * ·y+F * ;

[0109] In the formula, c* = [A*, B*, C*, D*, E*, F*] is the optimal fitting parameter; (x, y, z) is the coordinate of the depth of the fitting surface.

[0110] Step C, processing the gray image of the smaller area using the instance segmentation neural network model to obtain a corresponding mask image.

[0111] Step D, according to the defect edge on the mask image, the sub-point cloud data of the defect edge is extracted from the point cloud data of the smaller area; according to the sub-point cloud data, the perimeter, diameter and volume of the defect can be calculated.

[0112] Step E, difference calculation is performed between the sub-point cloud data and the smooth surface model to obtain the depth deviation of the defect, and then the actual depth of the defect is determined, and finally the geometric information of the defect is obtained.

[0113] The calculation expression of the depth deviation △D of the defect is: △D = P valid -fit z .

[0114] Figure 3 The flowchart of the embodiment of the image-based engine blade defect automatic detection method of the application.

[0115] As Figure 3 shown, the embodiment of the image-based engine blade defect automatic detection method of the application includes moving the blade to the detection site by using the robot gripper and taking a photo of the blade at the detection site by using a two-dimensional camera and a three-dimensional camera, and further includes steps (1) to (5), which are specifically as follows:

[0116] Step (1), taking a photo of the blade gripped by the robot gripper by using the two-dimensional camera to obtain a plurality of two-dimensional images covering the surface of the blade.

[0117] Preferably, the surface of the blade is divided into a plurality of detection points, and the gripper moves these detection points to the imaging center of the two-dimensional camera for photographing. The robot gripper grips the same blade twice, and the positions of the two grips are different, so that the surface covered by the gripper during the first grip can also be detected, so that the obtained two-dimensional image covers all key areas and avoids blind areas.

[0118] Preferably, the plurality of two-dimensional images includes furnace batch number images and local position images; wherein, for the furnace batch number images, the recognition of the furnace batch number is carried out using an OCR algorithm processing; for the local position images, the defect detection is carried out using a target detection neural network model, whereby the recognized furnace batch number will be bound with subsequent detection data (including defect detection results, quantitative data, etc.), ensuring that all detection information can be traced back to a specific production batch. Wherein, the OCR algorithm adopts DBM (Deep Bidirectional Model), uses bidirectional modeling technology for feature extraction and sequence modeling, captures the context relationship between characters, can deep feature learning to improve recognition accuracy and robustness, has strong generalization ability to adapt to various fonts and backgrounds, uses efficient training and reasoning mechanism to ensure fast processing of image data, so as to realize high-precision recognition under complex conditions.

[0119] Step (2), using a target detection neural network model to detect defects in the plurality of two-dimensional images.

[0120] The architecture of the target detection neural network model includes a backbone network, a neck network and a detection head, the backbone network is responsible for extracting image features, the neck network further processes and fuses these features, and the detection head generates the final detection results. The target detection neural network model adopts a continuous double assignment strategy without NMS (non-maximum suppression) training, which significantly improves the detection accuracy of small size and overlapping defects, can accurately identify small size defects, and effectively handles overlapping target problems, enhances the adaptability to complex background and high density targets. In addition, the NMS-free strategy reduces the possibility of false positives and false negatives, ensures the accuracy and stability of the detection results, making it more suitable for efficient and accurate detection of blade defects.

[0121] Step (3), using the above-mentioned calculation method of obtaining the center coordinates of the defects from the two-dimensional images, for the two-dimensional images detected with defects, calculating the three-dimensional coordinates of the defect center in the robot base coordinate system, and then moving the defect center to the imaging center of the three-dimensional camera by the gripper.

[0122] Step (4), acquiring the gray-scale image and point cloud data of the defect by the three-dimensional camera.

[0123] Step (5), using the above-mentioned defect quantification method based on the fusion of three-dimensional camera and neural network to process the gray-scale image and point cloud data of the defect, obtaining the geometric information of the defect.

[0124] Figure 4 A perspective view of an embodiment of the engine blade defect detection system of the present application. Figure 5 A top view of an embodiment of the engine blade defect detection system of the present application.

[0125] As shown in Figures 4-5 the engine blade defect detection system includes a tray 100, a robot 200, a two-dimensional camera 300, a three-dimensional camera 400, a light supplement source, a terminal device and a test bench 600. The surface of the test bench 600 is assembled by T-shaped grooves. The tray 100 is used to store blades, and the tray 100 is provided with blade storage grooves.

[0126] The robot 200 is used to clamp and move the blades. The robot 200 has a clamping jaw 210, a mechanical arm 220 and a base 230. The mechanical arm 220 can drive the blades to rotate and lift. The base 230 is detachably connected with the T-shaped grooves and can slide horizontally along the T-shaped grooves.

[0127] The two-dimensional camera 300 and the three-dimensional camera 400 are used to take pictures of the blades clamped by the clamping jaw 210. The three-dimensional camera 400 is a monocular structured light camera. The two-dimensional camera 300 is arranged in front of the robot 200, the tray 100 is arranged beside the robot 200, and the three-dimensional camera 400 is arranged between the tray 100 and the two-dimensional camera 300 and is arranged obliquely towards the robot 200.

[0128] The light supplement source is used to supplement light when the two-dimensional camera 300 and the three-dimensional camera 400 take pictures. The light supplement source includes a strip light source 510 and a ring light source 520. The ring light source 520 is arranged in front of the two-dimensional camera 300, and the strip light source 510 is arranged between the two-dimensional camera 300 and the robot 200. The strip light source 510 is at least two.

[0129] The two-dimensional camera 300, the three-dimensional camera 400 and the light supplement source are fixed on the test bench 600 through adjustable supports. The adjustable supports include a U-shaped base 710, a stand 720 and a hinged seat 730. The U-shaped base 710 is detachably connected with the T-shaped grooves and can slide horizontally along the T-shaped grooves. The two-dimensional camera 300, the three-dimensional camera 400 and the light supplement source are detachably connected with the stand 720 through the vertical through holes of the hinged seat 730 and can slide vertically and rotate horizontally along the stand 720. Further, the hinged seat 730 connected with the two-dimensional camera 300 or the three-dimensional camera 400 also includes a horizontal through hole. The adjustable supports also include a crossbar 740, which is detachably connected with the stand 720 through the horizontal through hole of the hinged seat 730 and can move along the horizontal through hole. The end of the crossbar 740 is connected with the two-dimensional camera 300 or the three-dimensional camera 400.

[0130] The terminal device communicates with the two-dimensional camera 300, the three-dimensional camera 400 and the robot 200 through a wireless network and controls the clamping and movement of the blades by the clamping jaw 210.

[0131] The beneficial effects of the present application are illustrated below by specific test data. The preferred technical solution with small error and higher efficiency is adopted, i.e. the three-dimensional coordinates of the defect center are directly obtained from the center point coordinates of the rectangular coordinate frame by T camera-to-gripper , T base-to-gripper , T gripper-to-base transformation, and the three-dimensional coordinates for calculating the defect size are obtained from the four corner point coordinates of the rectangular coordinate frame by T camera-to-gripper transformation. The specific calculation process and data are shown as follows:

[0132] 1. The parameters of the two-dimensional camera are calculated as follows:

[0133] Intrinsic matrix

[0134] Distortion coefficient d = [0.20334415671740425, 4.904385112163188, -0.0022254112636877267, 1.4572813224837689e-05, -1037.0344399454984];

[0135] Reprojection error = 0.047132498496030244;

[0136]

[0137] 2. The actual three-dimensional camera coordinates of the two-dimensional pixel coordinate matrix in the two-dimensional camera coordinate system are calculated as follows:

[0138] The target detection neural network model is used to detect the defects in the two-dimensional image and output the rectangular coordinate frame of the defects, and the four corner point coordinates of the rectangular coordinate frame are [1136.0, 1818.0], [1186.0, 1818.0], [1136.0, 1872.0], [1186.0, 1872.0], and the center point coordinates are [1161.0, 1845.0];

[0139] The actual three-dimensional camera coordinates of the four corner point coordinates in the two-dimensional camera coordinate system are calculated as follows:

[0140] The three-dimensional coordinates corresponding to [1136.0, 1818.0] are [-14.37580796, -5.92899294, 270.5];

[0141] The three-dimensional coordinates corresponding to [1186.0, 1818.0] are [-13.83966873, -5.92899294, 270.5];

[0142] [1136.0, 1872.0] corresponding three-dimensional coordinates: [-14.37580796, -5.34946993, 270.5];

[0143] [1186.0, 1872.0] corresponding three-dimensional coordinates: [-13.83966873, -5.34946993, 270.5];

[0144] The actual three-dimensional camera coordinates of the center point coordinates in the two-dimensional camera coordinate system are calculated as follows:

[0145] [-14.10773834, -5.63923143, 270.5].

[0146] 3. The three-dimensional coordinates of the center point coordinates under the robot base coordinate system are calculated as follows:

[0147] Through T camera-to-base The center point coordinates are converted into three-dimensional coordinates under the robot base coordinate system: [-631.83898003,

[0148] -128.41387102, 254.0063209].

[0149] 4. The three-dimensional coordinates of the center point coordinates under the robot gripper coordinate system are calculated as follows:

[0150]

[0151] Through T base-to-gripper The three-dimensional coordinates of the center point coordinates under the robot base coordinate system are converted into three-dimensional coordinates under the robot gripper coordinate system: [-3.98719122, 37.2480781, 36.18581824].

[0152] 5. The three-dimensional coordinates of the center point coordinates under the robot base coordinate system are calculated as follows:

[0153]

[0154] Through T gripper-to-base The three-dimensional coordinates of the center point coordinates under the robot gripper coordinate system are converted into three-dimensional coordinates under the robot base coordinate system: [-651.80876029, 255.67971206, 319.5136791].

[0155] 6. The gripper holds the blade to [-651.80876029, 255.67971206, 319.5136791], and then a three-dimensional camera is used to collect a large area of gray-scale image and Figure 6 point cloud data image as shown.

[0156] 7. Calculate the defect size as follows:

[0157] By T camera-to-base Convert the four corner point coordinates into three-dimensional coordinates in the robot base coordinate system, respectively:

[0158] [1136.0, 1818.0] corresponding three-dimensional coordinates: [-631.83906977, -128.14745721, 254.29644636];

[0159] [1186.0, 1818.0] corresponding three-dimensional coordinates: [-631.84835501, -128.68251816, 254.2949805];

[0160] [1136.0, 1872.0] corresponding three-dimensional coordinates: [-631.82960527, -128.14523899, 253.71765873];

[0161] [1186.0, 1872.0] corresponding three-dimensional coordinates: [-631.83889034, -128.68031196, 253.71616966];

[0162] Due to slight errors in the conversion data, the three-dimensional coordinates of the four corner points after compensation and lengthening and widening by 1mm are:

[0163] [1136.0, 1818.0] corresponding three-dimensional coordinates: [-631.83906977, -127.14745721, 255.29644636];

[0164] [1186.0, 1818.0] corresponding three-dimensional coordinates: [-631.84835501, -129.68251816, 255.2949805];

[0165] [1136.0, 1872.0] corresponding three-dimensional coordinates: [-631.82960527, -127.14523899, 252.71765873];

[0166] [1186.0, 1872.0] corresponding three-dimensional coordinates: [-631.83889034, -129.68031196, 252.71616966];

[0167] The length, width and height of the defect are calculated as follows: length ≈ 0.0187mm, width ≈ 2.5372mm, height ≈ 2.5802mm.

[0168] 8. According to the length, width and height of the defect, the gray-scale image and the point cloud data image of the larger area are cut to obtain the gray-scale image and the point cloud data image of the smaller area, respectively as shown in Figure 7 and Figure 8 .

[0169] 9. The point cloud data of the cut smaller area is subjected to surface fitting to obtain a smooth surface model as shown in Figure 9 .

[0170] 10. The gray-scale image of the smaller area is processed using an instance segmentation neural network model to obtain a mask map as shown in Figure 10 .

[0171] 11. According to the defect edge on the mask map, the sub-point cloud data image of the defect edge is extracted from the point cloud data image of the smaller area as shown in Figure 11 .

[0172] 12. The sub-point cloud data is subjected to difference calculation with the smooth surface model, and finally the type of the defect is obtained as a pit with a pit diameter of 0.549 mm and a pit depth of -0.158 mm, and the judgment result is unqualified.

[0173] The above describes the relevant content of the present application. Those skilled in the art will be able to implement the present application based on these descriptions. Based on the above content of the present application, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

Claims

1. An engine blade defect detection method, characterized in that: the robot's gripper holds the blade, a two-dimensional camera is used to take pictures of the blade to obtain multiple two-dimensional images covering the surface of the blade; a target detection neural network model is used to process the multiple two-dimensional images and filter out two-dimensional images containing defects; the gripper is moved to move the defect center to the imaging center of the three-dimensional camera to take pictures of the defect, obtaining a grayscale image and point cloud data of the defect; the grayscale image and point cloud data of the defect are processed to obtain the geometric information of the defect; wherein the method for calculating the coordinates of the defect center comprises the following steps: Step 100, multiply the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image with the intrinsic matrix of the two-dimensional camera to calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system; Step 200, according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera, the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system are calculated; Step 300, the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are converted into three-dimensional coordinates in the robot base coordinate system by using the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system; Step 400, the three-dimensional coordinates of the defect in the robot base coordinate system are converted into three-dimensional coordinates in the robot gripper coordinate system by using the transformation matrix from the robot base coordinate system to the robot gripper coordinate system; Step 500, the three-dimensional coordinates of the defect in the robot gripper coordinate system are subtracted from the origin of the robot gripper coordinate system to calculate a vector representing the relative position of the defect in the robot gripper coordinate system; Step 600, the vector is converted into three-dimensional coordinates of the defect in the robot base coordinate system by using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, and then the three-dimensional coordinates of the defect center in the robot base coordinate system are obtained.

2. The method of claim 1, wherein: a target detection neural network model is used to detect defects in two-dimensional images and output rectangular coordinate frames of the defects, the two-dimensional pixel coordinate matrix includes four corner point coordinates and a center point coordinate of the rectangular coordinate frame; firstly, the four corner point coordinates are converted into four three-dimensional coordinates in the robot base coordinate system, and then the three-dimensional coordinates of the defect center are calculated from the four three-dimensional coordinates; or, the three-dimensional coordinates of the defect center are directly obtained by converting the center point coordinates.

3. The method of claim 2, wherein: In step 300, the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system includes the following steps: acquire images of the checkerboard grabbed by the robot gripper at different poses through the two-dimensional camera; obtain the pixel coordinates of the corner points of the checkerboard; establish a mapping relationship between the pixel coordinates of the corner points and the robot base coordinate system; solve the intrinsic matrix, extrinsic matrix and distortion coefficient of the two-dimensional camera; minimize the re-projection error between the actual detected corner point position and the theoretical corner point position; optimize the intrinsic matrix, distortion coefficient and extrinsic matrix of the two-dimensional camera; apply the intrinsic matrix and extrinsic matrix to the conversion of the pixel coordinates of the corner points to obtain the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system.

4. The engine blade defect detection method of claim 3, characterized in that: The corner pixel coordinates P of the checkerboard are extracted from each 2D image I using the cv2.findChessboardCorners function of OpenCV gray,k uv,k = (x uv,k ,y uv,k ), where u and v denote the position of the corner on the checkerboard and k denotes the index of the image;​ The mapping relationship between the pixel coordinates of the corner points and the robot base coordinate system is established using Zhang's calibration method, wherein it is assumed that the three-dimensional coordinates of the checkerboard in the robot base coordinate system are P uv =(X uv , Y uv , Z uv ), wherein Z uv =0; The intrinsic matrix, extrinsic matrix and distortion coefficients of the two-dimensional camera are calculated using the calibrate Camera function of OpenCV, denoted as: where K is an intrinsic matrix, f x and f y are focal lengths, c x and c y are principal point coordinates; H k is a homography matrix; R k is a rotation matrix, T k is a translation vector, [R k | T k ] is a 3x4 matrix representing the combination of the rotation matrix and the translation vector, i.e., an extrinsic matrix; d is a distortion coefficient, k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients; The minimization of the re-projection error between the actual detected corner position and the theoretical corner position is denoted as: Minimize∑ k ∑ u,v ||P uv,k -K(R k X uv +T k )| 2 ; The intrinsic matrix, distortion coefficients and extrinsic matrix of the two-dimensional camera are optimized using the maximum likelihood estimation method; The intrinsic matrix and the extrinsic rectangle are applied to the conversion of the corner pixel coordinates, and a transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system is obtained, represented as:

5. The method of claim 4, wherein: In step 300, the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are denoted as: The three-dimensional coordinates of the defect in the robot base coordinate system are denoted as: wherein Z 2D (u, v) are the two-dimensional pixel coordinates of the defect, for testing the distance between the object and the two-dimensional camera.

6. The method of claim 5, wherein: In step 400, the three-dimensional coordinates of the defect in the robot gripper coordinate system are denoted as: In the formula, T base-to-gripper is a transformation matrix of the robot base coordinate system to the robot gripper coordinate system.

7. The method of claim 6, wherein: Step 500: vector of the relative position of the defect in the robot gripper coordinate system is represented as: wherein are unit vectors along the x, y, z axes, respectively.

8. The method of claim 7, wherein: In step 600, the three-dimensional coordinates of the defect center in the robot base coordinate system are denoted as: In the formula, T gripper-to-base is a transformation matrix of the robot gripper coordinate system to the robot base coordinate system.

9. The method of claim 8, wherein: The robot gripper is moved to the imaging center of the three-dimensional camera without clamping the test object, and the coordinates of the robot gripper are acquired through three-dimensional camera imaging, that is, the transformation matrix T of the imaging center of the three-dimensional camera in the gripper coordinate system and the robot base coordinate system is established gripper-to-base .

Images