Calculation method for obtaining defect center coordinate from two-dimensional image and application thereof

Abstract

The invention belongs to the technical field of image processing, and discloses a calculation method for obtaining a defect center coordinate from a two-dimensional image and application thereof, and the method comprises the following steps: step 100, calculating to obtain a normalized plane coordinate of a defect in a two-dimensional camera coordinate system; step 200, calculating an actual three-dimensional camera coordinate of the defect in the two-dimensional camera coordinate system; step 300, converting the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system into three-dimensional coordinates in the robot base coordinate system; step 400, converting the three-dimensional coordinates of the defect in the robot base coordinate system into three-dimensional coordinates in the robot clamping jaw coordinate system; step 500, subtracting the three-dimensional coordinate of the defect in the robot clamping jaw coordinate system from the original point of the robot clamping jaw coordinate system, and calculating a vector representing the relative position of the defect in the robot clamping jaw coordinate system; and step 600, converting the vector into a three-dimensional coordinate of the defect in the coordinate system of the robot base to obtain a three-dimensional coordinate of the center of the defect in the coordinate system of the robot base.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a calculation method for obtaining defect center coordinates from a two-dimensional image and an application thereof. Background Art

[0002] Aerospace engines are key power devices for aircraft and are known as the "heart" of aircraft and rockets. Their independent research and development is of great significance to the country's national defense construction, energy security and sustainable development of the environment. Among them, blades are the core components of aerospace engines that guide airflow and generate power. The detection of their key quality characteristics directly affects the service safety and reliability of aircraft.

[0003] Among the many parts of an engine, blades are numerous, accounting for about one-third of the total manufacturing volume. Blades are numerous, complex in shape, require high precision, are difficult to process, and are prone to failure. They are key components of an engine, which relies on many blades to compress and expand gas to generate powerful power to propel the aircraft forward. Therefore, the quality of blades is a key factor affecting the energy conversion efficiency and working performance of the engine.

[0004] At present, the main means of detecting blade surface defects is manual visual inspection. From the perspective of detection quality, the detection results are greatly affected by human factors. From the perspective of detection efficiency, the blade appearance is complex, there are many detection items, and the efficiency of manual detection is low. From the perspective of detection data, manual detection results are only qualitative judgments and lack quantitative data. The results are difficult to store digitally, which is not conducive to the traceability of quality issues in the later stage. From the perspective of detection quantity, there is still a lack of intelligent detection capabilities for batch blade surface defects on site. The method of relying on manual visual inspection has reached a production capacity bottleneck and is difficult to meet the needs of production and delivery. Summary of the invention

[0005] The main purpose of the present invention is to provide an image-based automatic engine blade defect detection method and an engine blade defect detection system, which can detect blade surface defects in a digital, automated and intelligent manner to achieve efficient and intelligent detection with stable quality, high efficiency, high traceability and quantified detection results.

[0006] In order to achieve the above-mentioned purpose, the technical solutions of the image-based automatic detection method of engine blade defects and the engine blade defect detection system provided by the present invention are as follows:

[0007] The image-based automatic detection method for engine blade defects includes using a robot gripper to clamp the blade and move it to a detection position, and using a two-dimensional camera or a three-dimensional camera to take a picture of the blade at the detection position, and also includes the following steps:

[0008] Step (1), photographing the blade gripped by the robot gripper by means of a two-dimensional camera to obtain a plurality of two-dimensional images covering the surface of the blade;

[0009] Step (2), using a target detection neural network model to perform defect detection on the multiple two-dimensional images;

[0010] Step (3), for the two-dimensional image of the detected defect, calculate the three-dimensional coordinates of the defect center in the robot base coordinate system, and then make the gripper move the defect center to the imaging center of the three-dimensional camera;

[0011] Step (4), collecting grayscale images and point cloud data of defects through a three-dimensional camera;

[0012] Step (5), using a defect quantification method based on the fusion of a three-dimensional camera and a neural network to process the grayscale image and point cloud data of the defect to obtain the geometric information of the defect.

[0013] The engine blade defect detection system comprises: a material tray for storing blades; a blade storage tank is provided on the material tray; a robot is used to clamp and move blades; the robot has a clamp and a mechanical arm; a two-dimensional camera is used to take pictures of the blades clamped by the clamp; a three-dimensional camera is used to take pictures of the blades clamped by the clamp; and a fill light source is used to provide fill light when the two-dimensional camera and the three-dimensional camera take pictures.

[0014] In the above-mentioned image-based automatic detection method for engine blade defects, when step (3) and step (5) adopt the following calculation method for obtaining the defect center coordinates in the two-dimensional image and the defect quantification method based on the fusion of three-dimensional camera and neural network, the calculation method is simple and has high calculation efficiency and accuracy.

[0015] To this end, the present invention provides a calculation method for obtaining the coordinates of the defect center from a two-dimensional image and a defect quantification method based on the fusion of a three-dimensional camera and a neural network. The technical solution is as follows:

[0016] A calculation method for obtaining the coordinates of the defect center from a two-dimensional image is used to enable the robot's gripper to grip the test object and move the defect center to the imaging center of the three-dimensional camera. The calculation method includes the following steps:

[0017] Step 100, multiplying the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image by the internal parameter matrix of the two-dimensional camera to calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system;

[0018] Step 200, calculating the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera;

[0019] Step 300, using a transformation matrix from a two-dimensional camera coordinate system to a robot base coordinate system, converting the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system into three-dimensional coordinates in the robot base coordinate system;

[0020] Step 400, using a transformation matrix from the robot base coordinate system to the robot gripper coordinate system, converting the three-dimensional coordinates of the defect in the robot base coordinate system into the three-dimensional coordinates in the robot gripper coordinate system;

[0021] Step 500, subtracting the three-dimensional coordinates of the defect in the robot gripper coordinate system from the origin of the robot gripper coordinate system, and calculating a vector representing the relative position of the defect in the robot gripper coordinate system;

[0022] Step 600, using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, convert the vector into the three-dimensional coordinates of the defect in the robot base coordinate system, and then obtain the three-dimensional coordinates of the defect center in the robot base coordinate system. The robot gripper can move the defect center to the imaging center of the three-dimensional camera.

[0023] The defect quantification method based on the fusion of three-dimensional camera and neural network includes the following steps:

[0024] Step A, cropping the grayscale image and point cloud data of a larger area collected by the 3D camera into a smaller area related to the defect, thereby obtaining the grayscale image and point cloud data of the smaller area;

[0025] Step B, performing surface fitting on the point cloud data of the smaller area after clipping to obtain a smooth surface model;

[0026] Step C, using the instance segmentation neural network model to process the grayscale image of a smaller area to obtain a corresponding mask image;

[0027] Step D, extracting sub-point cloud data of the defect edge from the point cloud data of a smaller area according to the defect edge on the mask image;

[0028] In step E, the geometric information of the defect can be obtained by performing difference calculation between the sub-point cloud data and the smooth surface model.

[0029] The calculation method for obtaining the defect center coordinates in the above-mentioned two-dimensional image and the defect quantification method based on the fusion of three-dimensional camera and neural network can be applied to not only engine blades and their defects, but also target areas of other test objects. The target area is not necessarily a defect, but can also be a functional structure specially processed. The defect is not necessarily a pit, but can also be a crack, convex spot and other defects.

[0030] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. Additional aspects and advantages of the present invention will be partially given in the following description, partially become apparent from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present invention are used to assist the understanding of the present invention. The contents provided in the drawings and their related descriptions in the present invention can be used to explain the present invention, but do not constitute improper limitations on the present invention.

[0032] Figure 1 The present invention is a flow chart of an embodiment of a method for calculating the coordinates of a defect center from a two-dimensional image.

[0033] Figure 2 It is a flow chart of an embodiment of the defect quantification method based on the fusion of three-dimensional camera and neural network of the present invention.

[0034] Figure 3 The present invention is a flowchart of an embodiment of an automatic detection method for engine blade defects based on images.

[0035] Figure 4 It is a stereoscopic diagram of an embodiment of an engine blade defect detection system of the present invention.

[0036] Figure 5 It is a top view of an embodiment of the engine blade defect detection system of the present invention.

[0037] Figure 6 The present invention uses a three-dimensional camera to acquire a grayscale image of a larger area.

[0038] Figure 7 The present invention uses a three-dimensional camera to collect point cloud data images of a larger area.

[0039] Figure 8 It is a grayscale image of a smaller area obtained by cutting out a grayscale image of a larger area in the present invention.

[0040] Fig. 9 The point cloud data image of a smaller area is obtained by cutting the point cloud data image of a larger area according to the present invention.

[0041] Fig.10 This is a smooth surface model image obtained by performing surface fitting on the point cloud data of a smaller area after clipping.

[0042] Fig.11 The present invention uses an instance segmentation neural network model to process a grayscale image of a smaller area to obtain a mask image.

[0043] Fig.12The present invention extracts a sub-point cloud data image of the defect edge from a point cloud data image of a smaller area.

[0044] The relevant marks in the above drawings are: 100-tray, 200-robot, 210-gripper, 220-mechanical arm, 230-base, 300-two-dimensional camera, 400-three-dimensional camera, 510-bar light source, 520-ring light source, 600-test bench, 710-U-shaped base, 720-column, 730-articulated base, 740-cross bar. DETAILED DESCRIPTION

[0045] The present invention is described clearly and completely below in conjunction with the accompanying drawings. A person skilled in the art will be able to implement the present invention based on these descriptions. Before describing the present invention in conjunction with the accompanying drawings, it should be particularly noted that:

[0046] The technical solutions and technical features provided in each part of the present invention, including the following description, may be combined with each other if there is no conflict.

[0047] In addition, the embodiments of the present invention involved in the following description are generally only a part of the embodiments of the present invention, rather than all the embodiments. Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] About the terms and units in the present invention: The terms "include", "have" and any variations thereof in the description and claims of the present invention and related parts are intended to cover non-exclusive inclusions.

[0049] Figure 1 The present invention is a flow chart of an embodiment of a method for calculating the coordinates of a defect center from a two-dimensional image.

[0050] like Figure 1 As shown, an embodiment of the calculation method for obtaining the coordinates of the defect center from a two-dimensional image of the present invention is used to enable the robot's gripper to clamp the test object and move the defect center to the imaging center of the three-dimensional camera. The calculation method includes steps 100 to 600, which are as follows:

[0051] Step 100: multiply the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image by the intrinsic parameter matrix of the two-dimensional camera to calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system.

[0052] Step 200 , calculating the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera.

[0053] Step 300, using the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system, convert the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system into the three-dimensional coordinates in the robot base coordinate system; wherein, the acquisition process of the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system includes steps 310 to 370, which are specifically as follows:

[0054] Step 310, collecting images of the chessboard grasped by the robot gripper in different positions through a two-dimensional camera;

[0055] Step 320, obtaining pixel coordinates of corner points of the chessboard;

[0056] Use OpenCV's cv2.find Chess board Corners function to find the Chess board Corners from each 2D image. gray,k Extract the pixel coordinates P of the corner points of the chessboard uv,k =(x uv,k ,y uv,k ), where u and v represent the positions of the corner points on the chessboard, and k represents the index of the image;

[0057] Step 330, establishing a mapping relationship between the corner point pixel coordinates and the robot base coordinate system;

[0058] The mapping relationship between the pixel coordinates of the corner points and the robot base coordinate system is established using Zhang Dingyou’s calibration method, where the three-dimensional coordinates of the chessboard in the robot base coordinate system are assumed to be P uv =(X uv , Y uv , Z uv ), where Z uv =0;

[0059] Step 340, solving the internal parameter matrix, external parameter matrix and distortion coefficient of the two-dimensional camera;

[0060] Use OpenCV's calibrate Camera function to calculate the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficient of the two-dimensional camera, which are expressed as follows:

[0061] H k =K·[R k |T k ]; d = [k1, k2, p1, p2, k3];

[0062] Where K is the internal parameter matrix, f x and f y is the focal length, c x and c y is the principal point coordinate; H k is the homography matrix; R kis the rotation matrix, T k is the translation vector, [R k |T k ] is a 3×4 matrix, which represents the combination of the rotation matrix and the translation vector, i.e., the extrinsic parameter matrix; d is the distortion coefficient, k1, k2, k3, k4 are the radial distortion coefficients, and p1, p2 are the tangential distortion coefficients;

[0063] Step 350, minimizing the reprojection error between the actual detected corner point position and the theoretical corner point position, is expressed as:

[0064] MinimizeΣ k ∑ u,v ||p uv,k -K(R k X uv +T k )|| 2 ;

[0065] Step 360, using the maximum likelihood estimation method to optimize the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix of the two-dimensional camera;

[0066] Step 370, applying the intrinsic parameter matrix and the extrinsic parameter matrix to the transformation of the corner pixel coordinates, and obtaining the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system is expressed as:

[0067] The actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are expressed as:

[0068]

[0069] The three-dimensional coordinates of the defect in the robot base coordinate system are expressed as:

[0070]

[0071] In the formula, Z 2D is the distance between the test object and the 2D camera, and (u,v) is the 2D pixel coordinate matrix of the defect.

[0072] Step 400, using a transformation matrix from the robot base coordinate system to the robot gripper coordinate system, converting the three-dimensional coordinates of the defect in the robot base coordinate system into the three-dimensional coordinates in the robot gripper coordinate system;

[0073] The three-dimensional coordinates of the defect in the robot gripper coordinate system are expressed as:

[0074]

[0075] Where, T base-to-gripper is the transformation matrix from the robot base coordinate system to the robot gripper coordinate system.

[0076] Step 500, subtracting the three-dimensional coordinates of the defect in the robot gripper coordinate system from the origin of the robot gripper coordinate system, and calculating a vector representing the relative position of the defect in the robot gripper coordinate system;

[0077] The vector of the relative position of the defect in the robot gripper coordinate system It is expressed as:

[0078]

[0079] In the formula, are unit vectors along the x, y, and z axes respectively.

[0080] Step 600, using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, the vector is converted into the three-dimensional coordinates of the defect in the robot base coordinate system, and then the three-dimensional coordinates of the defect center in the robot base coordinate system are obtained. The robot gripper can move the defect center to the imaging center of the three-dimensional camera;

[0081] The three-dimensional coordinates of the defect center in the robot base coordinate system (x base ,y base , z base ) is expressed as:

[0082]

[0083] Where, T gripper-to-base The acquisition method is as follows: when not holding the test object, move the robot gripper to the imaging center of the 3D camera, obtain the coordinates of the robot gripper through 3D camera imaging, and then establish the transformation matrix T of the 3D camera imaging center in the gripper coordinate system and the robot base coordinate system. gripper-to-base .

[0084] In the above-mentioned calculation method for obtaining the coordinates of the defect center from a two-dimensional image, a target detection neural network model is used to detect defects in a two-dimensional image and output a rectangular coordinate frame of the defect. The two-dimensional pixel coordinate matrix includes the coordinates of the four corner points and the center point of the rectangular coordinate frame. There are two ways to obtain the three-dimensional coordinates of the defect center in the robot base coordinate system, namely: (1) first calculate the coordinates of the four corner points through T camera-to-gripper , T base-to-gripper , T gripper-to-base Transformed into four 3D coordinates in the robot base coordinate system, and then the 3D coordinates of the defect center are calculated from the four 3D coordinates; (2) Directly calculate the 3D coordinates of the defect center from the center point coordinates through T camera-to-gripper , T base-to-gripper , T gripper-to-base The three-dimensional coordinates of the defect center are obtained by transformation. After verification, it is found that the second path has smaller error and higher efficiency.

[0085] Figure 2 It is a flow chart of an embodiment of the defect quantification method based on the fusion of three-dimensional camera and neural network of the present invention.

[0086] like Figure 2 As shown, an embodiment of the defect quantification method based on fusion of a three-dimensional camera and a neural network of the present invention includes steps A to E, which are specifically as follows:

[0087] Step A, according to the imaging center of the three-dimensional camera and the defect size calculated from the two-dimensional image, the grayscale image and point cloud data of the larger area collected by the three-dimensional camera are cropped into a smaller area related to the defect, so as to obtain the grayscale image and point cloud data of the smaller area; the three-dimensional camera adopts a monocular structured light camera;

[0088] The defect size is calculated by the four three-dimensional coordinates of the four corner points of the rectangular coordinate box of the defect output by the target detection neural network model processing the two-dimensional image in the robot base coordinate system, as follows:

[0089] Assume that the four three-dimensional coordinates of the four corner points in the robot base coordinate system are

[0090] Calculate the maximum and minimum values ​​of the four three-dimensional coordinates on the x-axis:

[0091]

[0092]

[0093] Calculate the maximum and minimum values ​​of the four three-dimensional coordinates on the y-axis:

[0094]

[0095]

[0096] Calculate the maximum and minimum values ​​of the four three-dimensional coordinates on the z-axis:

[0097]

[0098]

[0099] According to the maximum and minimum values ​​of the coordinate axis, calculate the length, width and height of the defect:

[0100] Length: length = max x -min x ;

[0101] Width: width = max y -min y;

[0102] Height: height = max z -min z ;

[0103] Among them, there are two paths for the four three-dimensional coordinates, namely: (1) sequentially passing the coordinates of the four corner points through T camera-to-gripper , T base-to-gripper , T gripper-to-base Then we get; (2) the coordinates of the four corner points are passed through T camera-to-gripper It is verified that the errors of the two paths are almost the same, but the efficiency of the second path is significantly higher.

[0104] Step B, performing surface fitting on the point cloud data of the cropped smaller area to obtain a smooth surface model; specifically includes the following steps B1 to B4:

[0105] Step B1, filter out valid point cloud data, and the calculation expression is: P valid =Mask·P raw Where, P valid is the effective point cloud data after screening; P raw It is the point cloud data of a smaller area; Mask is the mask obtained from the point cloud data collected by the 3D camera;

[0106] Step B2, performing surface fitting on the screened valid point cloud data to obtain a reference surface;

[0107] Step B3, using the BFGS algorithm in the minimization method to minimize the objective function of the fitting error, and obtain the optimal fitting parameters: wherein the expression of the objective function f(c) of the fitting error is:

[0108]

[0109] Where c = [A, B, C, D, E, F] is the fitting parameter; (x n ,y n , z n ) is the coordinate of the nth point cloud data.

[0110] Step B4, using the optimal fitting parameters to calculate the depth of the fitting surface, that is, to obtain a smooth surface model;

[0111] The calculation expression for calculating the depth z of the fitting surface using the optimal fitting parameters is:

[0112] fit z =A * ·x 2 +B * ·y 2 +C *·x·y+D * ·x+E * y+F * ;

[0113] In the formula, c*=[A*, B*, C*, D*, E*, F*] is the optimal fitting parameter; (x, y, z) is the coordinate at the depth of the fitting surface.

[0114] Step C: Use the instance segmentation neural network model to process the grayscale image of a smaller area to obtain the corresponding mask image.

[0115] Step D: extract sub-point cloud data of the defect edge from the point cloud data of a smaller area according to the defect edge on the mask image; and calculate the perimeter, diameter and volume of the defect according to the sub-point cloud data.

[0116] Step E, performing difference calculation between the sub-point cloud data and the smooth surface model to obtain the depth deviation of the defect, and then determine the actual depth of the defect, and finally obtain the geometric information of the defect;

[0117] The calculation expression of the defect depth deviation △D is: △D=P valid -fit z .

[0118] Figure 3 The present invention is a flowchart of an embodiment of an automatic detection method for engine blade defects based on images.

[0119] like Figure 3 As shown, an embodiment of the image-based automatic detection method for engine blade defects of the present invention includes using a robot gripper to clamp the blade and move it to a detection position, and using a two-dimensional camera or a three-dimensional camera to take a picture of the blade at the detection position, and also includes steps (1) to (5), which are specifically as follows:

[0120] Step (1) takes a picture of the blade gripped by the robot gripper by using a two-dimensional camera to obtain a plurality of two-dimensional images covering the surface of the blade.

[0121] Preferably, the blade surface is divided into multiple detection points, and the gripper accurately moves these detection points to the imaging center of the two-dimensional camera for taking pictures. The robot gripper grips the same blade twice, and the positions of the two grips are different, thereby ensuring that the surface covered by the gripper during the first grip can also be detected, so that the obtained two-dimensional image covers all key areas and avoids blind spots.

[0122] Preferably, the multiple two-dimensional images include furnace batch number images and local position images; wherein, for the furnace batch number images, the OCR algorithm is used to process the furnace batch number; for the local position images, the target detection neural network model is used to detect defects, thereby, the identified furnace batch number will be bound to the subsequent detection data (including defect detection results, quantitative data, etc.), ensuring that all detection information can be traced back to a specific production batch. Among them, the OCR algorithm adopts DBM (Deep Bidirectional Model), uses bidirectional modeling technology for feature extraction and sequence modeling, captures the contextual relationship between characters, can perform deep feature learning to improve recognition accuracy and robustness, has strong generalization ability to adapt to various fonts and backgrounds, and adopts efficient training and reasoning mechanisms to ensure rapid processing of image data, so that high-precision recognition can be achieved under complex conditions.

[0123] Step (2), using a target detection neural network model to perform defect detection on the multiple two-dimensional images.

[0124] The architecture of the target detection neural network model includes a backbone network, a neck network, and a detection head. The backbone network is responsible for extracting image features, the neck network further processes and fuses these features, and the detection head generates the final detection results. The target detection neural network model adopts a continuous dual allocation strategy without NMS (non-maximum suppression) training. This strategy significantly improves the detection accuracy of small-sized and overlapping defects, can accurately identify small-sized defects, and effectively handle overlapping target problems, and enhances the adaptability to complex backgrounds and high-density targets. In addition, the NMS-free strategy reduces the possibility of false detection and missed detection, ensures the accuracy and stability of the detection results, and makes it more suitable for efficient and accurate detection of blade defects.

[0125] Step (3) uses the above-mentioned calculation method for obtaining the coordinates of the defect center from the two-dimensional image, calculates the three-dimensional coordinates of the defect center in the robot base coordinate system for the two-dimensional image in which the defect is detected, and then enables the gripper to move the defect center to the imaging center of the three-dimensional camera.

[0126] Step (4), collecting the grayscale image and point cloud data of the defect through a three-dimensional camera.

[0127] Step (5), using the above-mentioned defect quantification method based on fusion of three-dimensional camera and neural network to process the grayscale image and point cloud data of the defect to obtain the geometric information of the defect.

[0128] Figure 4 It is a stereoscopic diagram of an embodiment of an engine blade defect detection system of the present invention. Figure 5 It is a top view of an embodiment of the engine blade defect detection system of the present invention.

[0129] like Figure 4-5 As shown, an embodiment of the engine blade defect detection system includes a tray 100, a robot 200, a two-dimensional camera 300, a three-dimensional camera 400, a fill light source, a terminal device and a test bench 600. The surface of the test bench 600 is assembled by T-shaped grooves. The tray 100 is used to store blades, and a blade storage tank is provided on the tray 100.

[0130] The robot 200 is used to clamp and move blades; the robot 200 has a clamp 210, a mechanical arm 220 and a base 230, the mechanical arm 220 can drive the blades to rotate and lift, and the base 230 is detachably connected to the T-slot and can slide horizontally along the T-slot.

[0131] The two-dimensional camera 300 and the three-dimensional camera 400 are used to take pictures of the blades clamped by the gripper 210. The three-dimensional camera 400 is a monocular structured light camera. The two-dimensional camera 300 is arranged in front of the robot 200, the material tray 100 is arranged on the side of the robot 200, and the three-dimensional camera 400 is arranged between the material tray 100 and the two-dimensional camera 300 and the shooting direction is arranged obliquely toward the robot 200.

[0132] The fill light source is used for fill light when the two-dimensional camera 300 and the three-dimensional camera 400 take pictures. The fill light source includes a bar light source 510 and a ring light source 520. The ring light source 520 is arranged in front of the two-dimensional camera 300, and the bar light source 510 is arranged between the two-dimensional camera 300 and the robot 200. There are at least two bar light sources 510.

[0133] The two-dimensional camera 300, the three-dimensional camera 400 and the fill-in light source are all fixed on the test bench 600 through an adjustable bracket. The adjustable bracket includes a U-shaped base 710, a column 720 and an articulated seat 730. The U-shaped base 710 is detachably connected to the T-slot and can slide horizontally along the T-slot. The two-dimensional camera 300, the three-dimensional camera 400 and the fill-in light source are detachably connected to the column 720 through the vertical through hole of the articulated seat 730 and can slide vertically and rotate horizontally along the column 720. Furthermore, the articulated seat 730 connected to the two-dimensional camera 300 or the three-dimensional camera 400 also includes a horizontal through hole. The adjustable bracket also includes a cross bar 740. The cross bar 740 is detachably connected to the column 720 through the horizontal through hole of the articulated seat 730 and can move along the horizontal through hole. The end of the cross bar 740 is connected to the two-dimensional camera 300 or the three-dimensional camera 400.

[0134] The terminal device communicates with the two-dimensional camera 300 , the three-dimensional camera 400 and the robot 200 via a wireless network and controls the gripper 210 to grip and move the blade.

[0135] The following is a detailed description of the beneficial effects of the present invention. camera-to-gripper , T base-to-gripper , T gripper-to-base The three-dimensional coordinates of the defect center are obtained by transformation, and the coordinates of the four corner points of the rectangular coordinate frame are converted through T camera-to-gripper The three-dimensional coordinates of the defect size are obtained. The specific calculation process and data are as follows:

[0136] 1. The parameters of the two-dimensional camera are calculated as follows:

[0137] Internal parameter matrix

[0138] Distortion coefficient d = [0.20334415671740425, 4.904385112163188, -0.0022254112636877267, 1.4572813224837689e-05, -1037.0344399454984];

[0139] Reprojection error = 0.047132498496030244;

[0140]

[0141] 2. Calculate the actual 3D camera coordinates of the 2D pixel coordinate matrix in the 2D camera coordinate system as follows:

[0142] The target detection neural network model is used to detect defects in the two-dimensional image and output the rectangular coordinate frame of the defect. The coordinates of the four corner points of the rectangular coordinate frame are: [1136.0, 1818.0], [1186.0, 1818.0], [1136.0, 1872.0], [1186.0, 1872.0], and the coordinates of the center point are [1161.0, 1845.0];

[0143] The actual three-dimensional camera coordinates of the four corner points in the two-dimensional camera coordinate system are calculated as follows:

[0144] The corresponding 3D coordinates of [1136.0, 1818.0] are: [-14.37580796, -5.92899294, 270.5];

[0145] [1186.0, 1818.0] corresponds to the 3D coordinates: [-13.83966873, -5.92899294, 270.5];

[0146] [1136.0, 1872.0] corresponds to the 3D coordinates: [-14.37580796, -5.34946993, 270.5];

[0147] The corresponding 3D coordinates of [1186.0, 1872.0] are: [-13.83966873, -5.34946993, 270.5];

[0148] The actual three-dimensional camera coordinates of the center point coordinates in the two-dimensional camera coordinate system are calculated as:

[0149] [-14.10773834, -5.63923143, 270.5].

[0150] 3. Calculate the three-dimensional coordinates of the center point in the robot base coordinate system as follows:

[0151] By T camera-to-base Convert the center point coordinates into three-dimensional coordinates in the robot base coordinate system: [-631.83898003,

[0152] -128.41387102,254.0063209].

[0153] 4. Calculate the three-dimensional coordinates of the center point in the robot gripper coordinate system as follows:

[0154]

[0155] By T base-to-gripper Convert the three-dimensional coordinates of the center point in the robot base coordinate system into the three-dimensional coordinates in the robot gripper coordinate system: [-3.98719122, 37.2480781, 36.18581824].

[0156] 5. Calculate the three-dimensional coordinates of the center point in the robot base coordinate system as follows:

[0157]

[0158] By T gripper-to-base Convert the three-dimensional coordinates of the center point in the robot gripper coordinate system into the three-dimensional coordinates in the robot base coordinate system: [-651.80876029, 255.67971206, 319.5136791].

[0159] 6. The gripper clamps the blade to [-651.80876029, 255.67971206, 319.5136791], and then uses a 3D camera to collect grayscale images and point cloud data images of a larger area, as shown in the figure. Figure 6 and Figure 7shown.

[0160] 7. Calculate the defect size as follows:

[0161] By T camera-to-base Convert the coordinates of the four corner points into three-dimensional coordinates in the robot base coordinate system, which are:

[0162] The corresponding 3D coordinates of [1136.0, 1818.0] are: [-631.83906977, -128.14745721, 254.29644636];

[0163] The corresponding 3D coordinates of [1186.0, 1818.0] are: [-631.84835501, -128.68251816, 254.2949805];

[0164] The corresponding 3D coordinates of [1136.0, 1872.0] are: [-631.82960527, -128.14523899, 253.71765873];

[0165] The corresponding 3D coordinates of [1186.0, 1872.0] are: [-631.83889034, -128.68031196, 253.71616966];

[0166] Since there is a slight error in the data during the conversion, the three-dimensional coordinates of the four corner points after compensation, lengthening and widening by 1mm are:

[0167] The corresponding 3D coordinates of [1136.0, 1818.0] are: [-631.83906977, -127.14745721, 255.29644636];

[0168] The corresponding 3D coordinates of [1186.0, 1818.0] are: [-631.84835501, -129.68251816, 255.2949805];

[0169] The corresponding 3D coordinates of [1136.0, 1872.0] are: [-631.82960527, -127.14523899, 252.71765873];

[0170] The corresponding 3D coordinates of [1186.0, 1872.0] are: [-631.83889034, -129.68031196, 252.71616966];

[0171] The length, width and height of the defect are calculated to be: length ≈ 0.0187 mm, width ≈ 2.5372 mm, height ≈ 2.5802 mm.

[0172] 8. According to the length, width and height of the defect, the grayscale image and point cloud data image of the larger area are cut to obtain the grayscale image and point cloud data image of the smaller area as follows: Figure 8 and Fig. 9 shown.

[0173] 9. Perform surface fitting on the point cloud data of the smaller area after cropping to obtain a smooth surface model such as Fig.10 shown.

[0174] 10. Use the instance segmentation neural network model to process the grayscale image of a smaller area, and the resulting mask image is as follows Fig.11 shown.

[0175] 11. According to the defect edge on the mask image, extract the sub-point cloud data image of the defect edge from the point cloud data image of a smaller area as shown in Fig.12 shown.

[0176] 12. The difference between the sub-point cloud data and the smooth surface model is calculated, and the final defect type is a pit, the pit diameter is 0.549mm, the pit depth is -0.158mm, and the judgment result is unqualified.

[0177] The above is a description of the relevant contents of the present invention. A person skilled in the art will be able to implement the present invention based on these descriptions. Based on the above contents of the present invention, all other embodiments obtained by a person skilled in the art without creative work shall fall within the scope of protection of the present invention.

Claims

1. A calculation method for obtaining the coordinates of the defect center from a two-dimensional image, which is used to enable the robot's gripper to hold the test object and move the defect center to the imaging center of a three-dimensional camera, characterized in that: The calculation method includes the following steps: Step 100, multiplying the two-dimensional pixel coordinate matrix of the defect in the two-dimensional image by the internal parameter matrix of the two-dimensional camera to calculate the normalized plane coordinates of the defect in the two-dimensional camera coordinate system; Step 200, calculating the actual three-dimensional camera coordinates of the defect in the two-dimensional camera coordinate system according to the normalized plane coordinates and the distance between the test object and the two-dimensional camera; Step 300, using a transformation matrix from a two-dimensional camera coordinate system to a robot base coordinate system, converting the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system into three-dimensional coordinates in the robot base coordinate system; Step 400, using a transformation matrix from the robot base coordinate system to the robot gripper coordinate system, converting the three-dimensional coordinates of the defect in the robot base coordinate system into the three-dimensional coordinates in the robot gripper coordinate system; Step 500, subtracting the three-dimensional coordinates of the defect in the robot gripper coordinate system from the origin of the robot gripper coordinate system, and calculating a vector representing the relative position of the defect in the robot gripper coordinate system; Step 600, using the transformation matrix from the robot gripper coordinate system to the robot base coordinate system, convert the vector into the three-dimensional coordinates of the defect in the robot base coordinate system, and then obtain the three-dimensional coordinates of the defect center in the robot base coordinate system. The robot gripper can move the defect center to the imaging center of the three-dimensional camera.

2. The method for calculating the coordinates of the defect center from a two-dimensional image according to claim 1, characterized in that: Using a target detection neural network model to detect defects in a two-dimensional image and output a rectangular coordinate frame of the defect, the two-dimensional pixel coordinate matrix including the coordinates of four corner points and a center point of the rectangular coordinate frame; First, the coordinates of the four corner points are converted into four three-dimensional coordinates in the robot base coordinate system, and then the three-dimensional coordinates of the defect center are calculated from the four three-dimensional coordinates; or, the three-dimensional coordinates of the defect center are directly converted from the center point coordinates.

3. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 2, characterized in that: In step 300, the process of acquiring the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system includes the following steps: The two-dimensional camera is used to capture images of the chessboard grasped by the robot gripper in different positions; Get the pixel coordinates of the corner points of the chessboard; Establish the mapping relationship between the corner point pixel coordinates and the robot base coordinate system; Solve the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient of the two-dimensional camera; Minimize the reprojection error between the actual detected corner positions and the theoretical corner positions; Optimize the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix of the 2D camera; The intrinsic parameter matrix and the extrinsic parameter matrix are applied to the conversion of the corner pixel coordinates to obtain the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system.

4. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 3, characterized in that: Use OpenCV's cv2.find Chess board Corners function to find the Chess board Corners from each 2D image. gray,k Extract the pixel coordinates P of the corner points of the chessboard uv,k =(x uv,k ,y uv,k ), where u and v represent the positions of the corner points on the chessboard, and k represents the index of the image; The mapping relationship between the pixel coordinates of the corner points and the robot base coordinate system is established using Zhang Dingyou’s calibration method, where the three-dimensional coordinates of the chessboard in the robot base coordinate system are assumed to be P uv =(X uv , Y uv , Z uv ), where Z uv =0; Use OpenCV's calibrate Camera function to calculate the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficient of the two-dimensional camera, which are expressed as follows: H k =K·[R k |T k ];d=[k1,k2,p1,p2,k3]; Where K is the internal parameter matrix, f x and f y is the focal length, c x and c y is the principal point coordinate; H k is the homography matrix; R k is the rotation matrix, T k is the translation vector, [R k |T k ] is a 3×4 matrix, which represents the combination of the rotation matrix and the translation vector, i.e., the external parameter matrix; d is the distortion coefficient, k1, k2, k3, k4 are the radial distortion coefficients, and p1, p2 are the tangential distortion coefficients; Minimizing the reprojection error between the actual detected corner point position and the theoretical corner point position is expressed as: Minimize k S u,v ||p uv,k -K(R k X uv +T k )|| 2 ; Use the maximum likelihood estimation method to optimize the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix of the two-dimensional camera; Applying the intrinsic parameter matrix and the extrinsic parameter rectangle to the transformation of the corner pixel coordinates, the transformation matrix from the two-dimensional camera coordinate system to the robot base coordinate system is expressed as:

5. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 4, characterized in that: In step 300, the actual three-dimensional coordinates of the defect in the two-dimensional camera coordinate system are expressed as: The three-dimensional coordinates of the defect in the robot base coordinate system are expressed as: In the formula, Z 2D is the distance between the test object and the 2D camera, and (u,v) is the 2D pixel coordinate of the defect.

6. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 5, characterized in that: In step 400, the three-dimensional coordinates of the defect in the robot gripper coordinate system are expressed as: Where, T base-to-gripper is the transformation matrix from the robot base coordinate system to the robot gripper coordinate system.

7. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 6, characterized in that: In step 500: the vector of the relative position of the defect in the robot gripper coordinate system It is expressed as: In the formula, are unit vectors along the x, y, and z axes respectively.

8. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 7, characterized in that: In step 600, the three-dimensional coordinates of the defect center in the robot base coordinate system are expressed as: Where, T gripper-to-base is the transformation matrix from the robot gripper coordinate system to the robot base coordinate system.

9. The calculation method for obtaining the defect center coordinates from a two-dimensional image according to claim 8, characterized in that: When the test object is not clamped, the robot gripper is moved to the imaging center of the 3D camera. The coordinates of the robot gripper are obtained through 3D camera imaging, and the transformation matrix T of the 3D camera imaging center in the gripper coordinate system and the robot base coordinate system can be established. gripper-to-base .

10. An engine blade defect detection method, characterized in that: The following steps are involved: A two-dimensional camera is used to take pictures of the leaf to obtain multiple two-dimensional images covering the surface of the leaf; Processing the plurality of two-dimensional images using a target detection neural network model and screening out two-dimensional images containing defects; Calculating the defect center coordinates using the calculation method described in any one of claims 1 to 9; Move the gripper to move the defect center to the imaging center of the 3D camera to take a picture of the defect, and obtain the grayscale image and point cloud data of the defect; The grayscale image and point cloud data of the defect are processed to obtain the geometric information of the defect.

Images