A computer vision-based 3D printing method for in-situ structural repair

Abstract

The present invention relates to the fields of computer vision and 3D printing, and more specifically to a computer vision-based in-situ repair 3D printing method for a structure, the method comprising the following steps: S100: placing a to-be-repaired filling structure in a calibration frame of a 3D printer base; S200: a camera on the 3D printer reads image information on the base; S300: performing computer vision-based operations on the image information to extract the contour of the to-be-repaired filling area; S400: selecting a filling algorithm according to the contour features of the to-be-repaired filling area, and generating a filling path for 3D printing repair; the method can avoid the cumbersome process limitation of first modeling, then manufacturing, and finally shifting and filling a damaged area, can directly perform rapid identification and accurate contour extraction on a damaged structure placed inside the printer, reconstruct and plan a printing path, and finally flexibly realize the in-situ repair and filling process of the damaged structure.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and 3D printing, and more specifically to a computer vision-based structure in-situ repair 3D printing method. Background Art

[0002] Designed thin plates or thin-walled structures can withstand greater loads with less weight and less material, and are widely used in aerospace, construction, vehicles and other fields or industries. The process of repairing a damaged structure is essentially a process of adding material and filling to the original structure; usually this process has a low degree of automation due to specific differences, is inefficient and mainly relies on manual work. 3D printing technology developed based on the principle of material addition and layer-by-layer accumulation can well meet the needs of structural repair work in terms of principle and implementation process, while also having high flexibility.

[0003] Current 3D printing technology mainly relies on inputting a pre-built 3D model into the printer, and works completely according to the trajectory path and process parameters in the set input model. For different contours of the structure to be repaired, it is necessary to first reconstruct an accurate virtual 3D model. This modeling process is complicated and it is difficult to place the reconstructed model on the original damaged structure to achieve in-situ printing and repair due to positioning accuracy issues.

[0004] At present, deep neural network models such as Mark R-CNN that use instance or semantic segmentation to extract the contour features of damaged structures in situ are relatively rough, which affects the printing quality, and the annotation process is time-consuming. If the damaged structure area is first quickly and roughly locked in combination with target recognition, and then image processing related operations are used in the locked area, it is expected to achieve fast and accurate positioning and extraction of the contour. Summary of the invention

[0005] The purpose of the present invention is to provide a computer vision-based 3D printing method for in-situ repair of structures, which can solve the problem that existing 3D printers are difficult to quickly identify damaged areas of damaged structures placed therein and accurately plan printing paths, and ultimately achieve in-situ repair and filling.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A computer vision-based in-situ structural repair 3D printing method, the method comprising the following steps:

[0008] S100: The filling structure to be repaired is placed in a calibration frame of the base of the 3D printer;

[0009] S200: The camera on the 3D printer reads the image information on the base;

[0010] S300: performing computer vision-based operations on the image information to extract the contour of the area to be repaired and filled;

[0011] S400: selecting a filling algorithm according to the contour features of the area to be repaired and generating a filling path for 3D printing repair;

[0012] The filling structure to be repaired is a thin plate, and the thickness of the filling area to be repaired is uniform;

[0013] The 3D printer is a material extrusion-based 3D printer, and its base is a black hot bed or a supporting plate. The origin of the 3D printer is used as the starting point, and a line segment with a length of 200 mm along the x and y directions of the printer is used as the side length to construct a square calibration frame;

[0014] The camera is movably mounted on the 3D printer, the angle between the optical axis or the central axis of the camera lens and the perpendicular line of the center point of the calibration frame is less than 15°, and the camera field of view includes the calibration frame;

[0015] The computer vision-based operations include deep neural network training and image processing for target recognition;

[0016] The specific process of performing computer vision-based operations on the image information includes the following steps:

[0017] S301: Obtain images of thin plates with different areas to be repaired and filled, perform data enhancement to obtain a training data set, annotate the damaged areas, train a deep neural network model using a two-stage algorithm or a one-stage algorithm, and complete a rough target detection model for the damaged areas to be filled; two-stage algorithms include R-CNN and Faster R-CNN, and one-stage algorithms include YOLOvX;

[0018] S302: obtaining an image with a resolution of 1280×960 through a camera, performing an opening operation on the image to remove image noise, wherein a kernel kernel uses a 3×3 matrix whose element values ​​are all 1, and the number of iterations is 2;

[0019] S303: using a watershed algorithm to identify and segment the thin plate, the foreground area of ​​the calibration frame, and the background area of ​​the 3D printer base;

[0020] In the foreground area extraction, the distance between each non-zero point in the noise-removed image and the nearest zero point is first calculated to obtain an image containing distance transformation information and perform image binarization processing, where the distance in the distance transformation is the Euclidean distance, and the threshold in the binarization is half of the maximum distance obtained in the distance transformation; then the watershed algorithm is used to obtain the foreground area containing the thin plate and the calibration frame, and the entire area is marked as white RGB (255,255,255); the background area is obtained by dilating the noise-removed image, and the background area is marked as black RGB (0,0,0);

[0021] In the image containing the foreground area, find each contour and calculate the area enclosed by the closed-loop contour, and select the contour line with the largest area as the contour line of the calibration frame; select and fit the points on the contour line of the calibration frame, where the maximum distance d between the fitted contour and the original contour is t for:

[0022] d t =0.1×l(c max )

[0023] Where l(c max ) is the maximum contour c max circumference.

[0024] Harris corner detection is performed on the obtained calibration frame fitting contour. The neighborhood size considered for corner detection is 4 pixels, the aperture window size used for Sobel derivative is 3 pixels, and the free parameter in the Harris corner detection equation is 0.03. Finally, the coordinate values ​​of the four corner points of the actual calibration frame in the image acquired by the camera are screened out.

[0025] S304: construct a calibration image with the same pixel size as the image acquired by the camera, and the pixel value RGB is (0,0,0); establish a virtual calibration frame in the calibration image, and the pixel coordinates of the four corners of the virtual calibration frame are (240,80), (1040,80), (240,880), (1040,880), that is, a 800×800 rectangle symmetrical along the center of the calibration image corresponds to a calibration frame of 200mm×200mm in actual size; according to the corner point coordinates in the actual calibration frame and the corner point coordinates in the virtual calibration frame, establish a perspective transformation matrix between the image taken by the camera and the calibration image;

[0026] S305: Performing Sobel filter transformation along the x and y directions on the image obtained by the camera, respectively, the filter derivative order is 1, and the length and width of the filter are 3 pixels; taking the absolute values ​​of the image pixel values ​​after the Sobel filter transformation in the two directions, and then assigning weights of 0.5 respectively and adding them at the corresponding positions to obtain a composite image with edge contour features; performing perspective transformation on the composite image using the perspective transformation matrix obtained in step S304, and obtaining a new image with edge contour features after position correction;

[0027] S306: After the image obtained in step S305 is converted into a grayscale image, image enhancement is performed: bilateral filtering is performed, and the diameter of the surrounding neighborhood is set to 60 pixels when operating each pixel, the sigma value of the color space filter is set to 10, and the sigma value of the filter in the sigmaSpace coordinate space is set to 100; then each pixel point of the obtained image is linearly enhanced so that the pixel value is doubled; the bilateral filtering operation is performed again, and this time the diameter of the surrounding neighborhood is set to 40 pixels when operating each pixel, and the sigma values ​​of the color space filter and the filter in the sigmaSpace coordinate space remain unchanged;

[0028] S307: Perform image recognition on the image obtained by the camera using the neural network trained by S301, obtain the target recognition rectangular frame of the damaged area, and obtain a new target recognition rectangular frame after perspective matrix transformation; perform contour extraction on the image in step S306 in the area surrounded by the new target recognition rectangular frame, and select the closed-loop contour with the largest area as the contour of the final area to be repaired and filled; for each pixel point coordinate (px, py) on the contour, perform the following conversion to obtain the real fill coordinate (x, y) to be printed:

[0029]

[0030] According to the contour features of the area to be repaired, a filling algorithm is selected, including a zig-zag parallel line filling algorithm and a contour line equidistant offset filling algorithm. The generated filling path is combined with process parameters such as extrusion volume and feed ratio to finally generate a complete printing code that can be recognized by the printer and input into the printer for in-situ online repair printing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] Figure 1 It is a schematic diagram of the flow of the computer vision-based in-situ structural repair 3D printing method of the present invention;

[0033] Figure 2 It is a schematic structural diagram of the in-situ repair 3D printing device of the present invention;

[0034] Figure 3 It is a schematic diagram of data annotation and building a deep neural network model of the present invention;

[0035] Figure 4 It is a schematic diagram of the training results of target recognition using the YOLOv5 neural network of the present invention;

[0036] Figure 5 It is a schematic diagram of image processing based on computer vision of image information of the present invention;

[0037] Figure 6 It is a schematic diagram of the 3D printing result of the in-situ repair of the structure according to the present invention.

[0038] In the figure: thin plate 1; 3D printer base 2; calibration frame 3; camera 4; area to be repaired and filled 5; controller and interface 6; FDM printer 7; corner point 8; virtual calibration frame 9; target recognition rectangular frame 10. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0040] like Figures 1 to 6 As shown, in order to solve the problem that it is difficult for existing 3D printers to quickly identify the damaged area of ​​the damaged structure placed therein and accurately plan the printing path, and finally realize in-situ repair and filling, the steps of a computer vision-based in-situ repair 3D printing method for the structure are described in detail below;

[0041] A computer vision-based in-situ structural repair 3D printing method, the method comprising the following steps:

[0042] S100: The filling structure to be repaired is placed in the calibration frame 3 of the 3D printer base 2;

[0043] S200: The camera 4 on the 3D printer reads the image information on the base 2;

[0044] S300: performing computer vision-based operations on the image information to extract the contour of the area 5 to be repaired and filled;

[0045] S400: selecting a filling algorithm according to the contour features of the filling area 5 to be repaired, and generating a filling path for 3D printing repair;

[0046] By establishing a neural network for target recognition, the area 5 to be repaired and filled is quickly screened out, and a series of image processing methods are used to accurately extract the contour boundary of the area 5 to be repaired and filled. At the same time, a calibration frame 3 is set in the printer, and the camera 4 can be used to realize the position calibration of the damaged structure flexibly placed inside the 3D printer, and establish a mapping relationship between the real coordinates and the visual image coordinate system. This method can perform in-situ repair and filling of different damaged thin plates 1, and does not require specialized designers to perform modeling operations based on the damaged structure, which reduces the operating threshold of the 3D printer and gets rid of the shape restrictions of the area 5 to be repaired and filled.

[0047] The filling structure to be repaired is a thin plate 1, that is, the ratio of the thickness of the thin plate 1 structure to the minimum dimension of the plate surface side length is between 0.01 and 0.1, the thickness of the filling area 5 to be repaired is uniform, and all of them are located in the calibration frame 3, and the thin plate 1 can also include other different integrated devices in addition to the filling area to be repaired;

[0048] The 3D printer is a material extrusion-based 3D printer, and its base 2 is a black hot bed or a supporting plate. Taking the origin of the 3D printer as the starting point, a square calibration frame 3 is constructed along line segments with a length of 200 mm in the x and y directions of the printer as the side lengths; the sides of the calibration frame 3 are white, and are affixed to the black base 2 with white stickers, or drawn on the black base 2 with a white marker, and the thickness of the frame is 1-3 mm.

[0049] The camera 4 is movably mounted on the 3D printer. The camera 4 can be connected to the 3D printer by a hinge or clearance fit method in the prior art, so as to facilitate the adjustment of the viewing angle for different damaged parts in any placement and lighting environment. The angle between the optical axis or the central axis of the camera 4 lens and the vertical line of the center point of the calibration frame 3 is less than 15°, and the field of view of the camera 4 includes the calibration frame 3 and can be clearly focused.

[0050] The computer vision-based operations include deep neural network training and image processing for target recognition;

[0051] The specific process of performing computer vision-based operations on the image information includes the following steps:

[0052] S301: Obtain images of a thin plate 1 with different areas 5 to be repaired and fill, and perform data enhancement to obtain a training data set; annotate the damaged area, train a deep neural network model using a two-stage algorithm or a one-stage algorithm, and complete a rough target detection model for the damaged area to be filled; finally obtain 450 training data set images and then annotate the targets, and train a deep neural network model using the YOLOv5 algorithm, such as Figure 3 The results obtained during the training process are shown in Figure 4As shown, the network can accurately complete the target detection of the area 5 to be repaired;

[0053] S302: obtaining an image with a resolution of 1280×960 through camera 4; performing an opening operation on the image to remove image noise, wherein the kernel kernel uses a 3×3 matrix whose element values ​​are all 1, and the number of iterations is 2;

[0054] S303: using a watershed algorithm to identify and segment the thin plate 1, the foreground area of ​​the calibration frame 3, and the background area of ​​the 3D printer base 2;

[0055] In the foreground area extraction, the distance transformation function distanceTransform in the OpenCV library is called to calculate the distance between each non-zero point in the image after noise removal and the nearest zero point. First, the distance between each non-zero point in the image after noise removal and the nearest zero point is calculated to obtain an image containing distance transformation information and perform image binarization processing, where the distance in the distance transformation is the Euclidean distance, and the threshold in the binarization is half of the maximum distance obtained in the distance transformation; then the watershed algorithm is used to obtain the foreground area containing the thin plate 1 and the calibration frame 3, and the whole is marked as white RGB (255,255,255); the background area is obtained by dilating the image after noise removal, and the background area is marked as black RGB (0,0,0);

[0056] In the image containing the foreground area, find each contour and calculate the area enclosed by the closed-loop contour, and select the contour line with the largest area as the contour line of the calibration frame 3; select and fit the points on the contour line of the calibration frame 3, where the maximum distance d between the fitted contour and the original contour is t for:

[0057] d t =0.1×l(c max )

[0058] Where l(c max ) is the maximum contour c max circumference.

[0059] Harris corner point 8 detection is performed on the obtained calibration frame 3 fitting contour, where the neighborhood size considered for corner point 8 detection is 4 pixels, the aperture window size used for Sobel derivative is 3 pixels, and the free parameter in the Harris corner point 8 detection equation is 0.03. Finally, the coordinate values ​​of the four corner points 8 of the actual calibration frame 3 in the image acquired by camera 4 are screened out;

[0060] S304: construct a calibration image with the same pixel size as the image acquired by camera 4, and the pixel value RGB is (0,0,0); establish a virtual calibration frame 9 in the calibration image, and the pixel coordinates of the four corners of the virtual calibration frame 9 are (240,80), (1040,80), (240,880), (1040,880), that is, a 800×800 rectangle symmetrical along the center of the calibration image, corresponding to the actual size of the calibration frame 3 of 200mm×200mm; according to the coordinates of the corner point 8 in the actual calibration frame 3 and the coordinates of the corner point 8 in the virtual calibration frame 9, establish the perspective transformation matrix between the image taken by camera 4 and the calibration image;

[0061] S305: Performing Sobel filter transformation along the x and y directions on the image obtained by camera 4, respectively, with the filter derivative order being 1, and the length and width of the filter being 3 pixels; taking the absolute values ​​of the image pixel values ​​after the Sobel filter transformation in the two directions, respectively assigning weights of 0.5 and adding them at corresponding positions to obtain a composite image with edge contour features; performing perspective transformation on the composite image using the perspective transformation matrix obtained in step S304, to obtain a new image with edge contour features after position correction;

[0062] S306: After the image obtained in step S305 is converted into a grayscale image, image enhancement is performed: bilateral filtering is performed, and the diameter of the surrounding neighborhood is set to 60 pixels when operating each pixel, the sigma value of the color space filter is set to 10, and the sigma value of the filter in the sigmaSpace coordinate space is set to 100; then each pixel point of the obtained image is linearly enhanced so that the pixel value is doubled; the bilateral filtering operation is performed again, and this time the diameter of the surrounding neighborhood is set to 40 pixels when operating each pixel, and the sigma values ​​of the color space filter and the filter in the sigmaSpace coordinate space remain unchanged;

[0063] S307: Perform image recognition on the image obtained by the camera 4 using the neural network trained by S301, obtain the target recognition rectangular frame 10 of the damaged area, and obtain a new target recognition rectangular frame 10 after perspective matrix transformation; perform contour extraction on the image in step S306 in the area surrounded by the new target recognition rectangular frame 10, and select the closed-loop contour with the largest area as the contour of the final area 5 to be repaired; for each pixel point coordinate (px, py) on the contour, perform the following conversion to obtain the real fill coordinate (x, y) to be printed:

[0064]

[0065] According to the contour features of the filling area 5 to be repaired, a filling algorithm is selected, and the filling algorithms include a zig-zag parallel line filling algorithm and a contour line equidistant offset filling algorithm. The filling algorithm is written in Python: the steps of the parallel line filling algorithm are to generate equally spaced parallel lines, and then intersect with the contour to obtain the intersection points, and then connect the intersection points in sequence to obtain the path; the contour line equidistant offset uses the clipper library library of graphics processing, calls the plane curve offset method based on Boolean operations to generate a plane offset curve, and performs the offset curve to obtain the corresponding path. The generated filling path is combined with process parameters such as extrusion volume and feed ratio to finally generate a complete G code that can be recognized by the printer, and input into the printer for the in-situ online repair printing process. Using the above-mentioned computer vision-based structural in-situ repair 3D printing method, the thin plate 1 structure containing the filling area 5 to be repaired is repaired in situ, and the result is finally obtained as shown Figure 6 shown.

Claims

1. A computer vision-based in-situ structural repair 3D printing method, characterized in that: The method comprises the following steps: S100: The structure to be repaired and filled is placed in a calibration frame (3) of a 3D printer base (2); S200: The camera (4) on the 3D printer reads image information on the base (2); S300: performing computer vision-based operations on the image information to extract the contour of the area to be repaired (5); S400: selecting a filling algorithm according to the contour features of the area to be repaired (5), and generating a filling path for 3D printing repair; The computer vision-based operation includes deep neural network training and image processing for target recognition, and step S300 specifically includes: S301: Acquire images of a thin plate (1) with different areas to be repaired and filled (5) and perform data enhancement to obtain a training data set, annotate the damaged areas, train a deep neural network model using a two-stage algorithm or a one-stage algorithm, and complete a rough target detection model for the damaged areas to be filled; S302: Obtain an image through the camera (4), perform an opening operation on the image to remove image noise, wherein the kernel uses a 3×3 matrix whose element values ​​are all 1, and the number of iterations is 2; S303: using a watershed algorithm to identify and segment the thin plate (1), the foreground area of ​​the calibration frame (3) and the background area of ​​the 3D printer base (2); in the image including the foreground area, finding each contour and calculating the area enclosed by the closed-loop contour, selecting the contour line with the largest area as the contour line of the calibration frame (3); selecting and contour fitting the points on the contour line of the calibration frame (3); wherein the maximum distance between the fitted contour and the original contour d t for: ; S304: constructing a calibration image with the same pixel size as the image acquired by the camera (4), with the pixel value RGB being (0,0,0); establishing a virtual calibration frame (9) in the calibration image; and establishing a perspective transformation matrix between the image captured by the camera (4) and the calibration image based on the coordinates of the corner point (8) in the actual calibration frame (3) and the coordinates of the corner point (8) in the virtual calibration frame (9); S305: The images obtained by the camera (4) are respectively processed along x and y Sobel filter transformation in two directions; taking the absolute values ​​of the image pixel values ​​after the Sobel filter transformation in two directions, assigning weights of 0.5 respectively and adding them at corresponding positions to obtain a composite image with edge contour features; performing perspective transformation on the composite image using the perspective transformation matrix obtained in step S304 to obtain a new image with edge contour features after position correction; S306: converting the image obtained in step S305 into a grayscale image and performing image enhancement; S307: Perform image recognition on the image obtained by the camera (4) using the neural network trained in S301 to obtain a target recognition rectangular frame (10) of the damaged area, and obtain a new target recognition rectangular frame (10) after perspective matrix transformation; perform contour extraction on the image in step S306 within the area enclosed by the new target recognition rectangular frame (10), and select the closed-loop contour with the largest area as the contour of the final area to be repaired and filled (5).

2. The computer vision-based in-situ structural repair 3D printing method according to claim 1, characterized in that: The filling structure to be repaired is a thin plate (1), and the thickness of the filling area (5) to be repaired is uniform.

3. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: The 3D printer is a material extrusion-based 3D printer, and its base (2) is a black hot bed or a supporting plate. The origin of the 3D printer is used as the starting point, and line segments with a length of 200 mm along the x and y directions of the printer are used as the side lengths to construct a square calibration frame (3).

4. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: The camera (4) is movably mounted on the 3D printer, the angle between the optical axis or the central axis of the camera (4) lens and the perpendicular line of the center point of the calibration frame (3) is less than 15°, and the field of view of the camera (4) includes the calibration frame (3).

5. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: In the step S303, the obtained calibration frame (3) fitting contour is subjected to Harris corner point (8) detection, wherein the neighborhood size considered for the corner point (8) detection is 4 pixels, the aperture window size of the Sobel derivative is 3 pixels, and the free parameter in the Harris corner point (8) detection equation is 0.03, and finally the coordinate values ​​of the four corner points (8) of the actual calibration frame (3) in the image acquired by the camera (4) are screened out; The pixel coordinates of the four corners of the virtual calibration frame (9) in S304 are (240, 80), (1040, 80), (240, 880), (1040, 880), respectively, i.e., a 800×800 rectangle symmetrical along the center of the calibration image, corresponding to the actual size of the calibration frame (3) of 200 mm×200 mm; In the step S305, the filter derivative order is 1, and the length and width of the filter are 3 pixels; The S306 also includes the following steps: Perform bilateral filtering, set the diameter of the neighborhood around each pixel to 60 pixels, set the sigma value of the color space filter to 10, and set the sigma value of the filter in the sigmaSpace coordinate space to 100; then linearly enhance each pixel of the obtained image to double the pixel value; perform bilateral filtering again, set the diameter of the neighborhood around each pixel to 40 pixels, and keep the sigma values ​​of the color space filter and the filter in the sigmaSpace coordinate space unchanged; The S307 also includes the following steps: For each pixel coordinate on the contour ( px , py ), and then the following transformation is performed to obtain the actual fill coordinates to be printed ( x , y ): 。 6. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: S303 in The maximum contour c max circumference.

7. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: The process of extracting the foreground area in S303 is as follows: first, the distance between each non-zero point in the noise-removed image and its nearest zero point is calculated to obtain an image containing distance transformation information and perform image binarization processing, wherein the distance in the distance transformation is the Euclidean distance, and the threshold in the binarization is half of the maximum distance obtained in the distance transformation; then, the watershed algorithm is used to obtain the foreground area containing the thin plate (1) and the calibration frame (3), and the foreground area is marked as white RGB (255,255,255) as a whole; the background area is obtained by performing an expansion transformation on the noise-removed image, and the background area is marked as black RGB (0,0,0).

8. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: The resolution of the image obtained by the camera (4) in S302 is 1280×960.

9. The computer vision-based in-situ structure repair 3D printing method according to claim 1, characterized in that: The filling algorithms include a zig-zag parallel line filling algorithm and a contour line equidistant offset filling algorithm.

Images