An automatic quantitative glue coating and plugging system based on 3D vision

Abstract

The application discloses an automatic quantitative glue coating and plugging system based on 3D vision, which comprises an industrial computer, a movable platform, a mechanical arm, a 3D camera, a plug stock bin assembly and a terminal device; the mechanical arm is provided with the 3D camera and the terminal device at the tail end; the terminal device comprises a pneumatic gripper assembly, a glue coating assembly and a glue scraping plate; when coating, the industrial computer controls the mechanical arm to drive the 3D camera to move to the top of a hole to be coated to obtain point cloud data of the hole to be coated, and processes the point cloud to obtain the center coordinates, the radius and the depth of the hole to be coated; the industrial computer calculates the glue coating amount according to the radius and the depth of the hole and the glue coating thickness; the plug stock bin assembly pops out a plug matched with the radius of the hole; after the industrial computer controls the pneumatic gripper to take out the popped plug, the industrial computer controls the glue coating assembly to move to the center coordinates to coat glue according to the calculated glue coating amount, and after coating, the industrial computer controls the pneumatic gripper to put the taken plug into the hole.

Description

Technical Field

[0001] This invention relates to the field of automatic adhesive application technology, and more particularly to an automatic quantitative adhesive application and sealing system based on 3D vision. Background Technology

[0002] During flight, aircraft surfaces experience significant temperature rises, necessitating the use of thermal protection materials to maintain structural temperatures within permissible ranges. Systems employing thermal protection materials require screws for installation, leaving cylindrical holes. To prevent screws from slipping out, these holes are sealed with adhesive plugs. Currently, this sealing process is typically manual. This manual method demands high operator skill and hinders standardization and regulation, limiting the improvement of sealing quality and production capacity.

[0003] Existing technologies also include several automated adhesive application methods, primarily two: robotic "teaching" adhesive application and vision-guided robotic adhesive application. Manual "teaching" involves a fixed multi-level adhesive application trajectory; if the product changes, the trajectory needs to be re-taught manually. Vision-guided robotic adhesive application can apply adhesive based on real-time image information, requiring no human intervention, saving time and effort. However, the key to vision-guided robotic adhesive application lies in the accuracy of the vision system's ability to identify the product to be coated.

[0004] Furthermore, there is a lack of existing research on automatic gluing for round holes in products, especially on the problem of automatic gluing and sealing of irregularly shaped round holes with uneven distribution. Research on calculating the amount of glue applied to irregularly shaped round holes is even rarer. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide an automatic quantitative adhesive application and sealing system based on 3D vision to solve the problem of automatic adhesive application to large batches of irregularly shaped circular holes with uneven distribution.

[0006] This invention provides an automated quantitative adhesive application and sealing system based on 3D vision, comprising:

[0007] Industrial control computer, mobile platform, robotic arm, 3D camera, end cap hopper assembly and end device;

[0008] The robotic arm is mounted on a movable platform, and a 3D camera and an end effector are installed at the end of the robotic arm; the industrial control computer is located inside the movable platform.

[0009] The end effector includes a pneumatic gripper assembly, an adhesive application assembly, and an adhesive scraper.

[0010] A plug hopper assembly is also placed on top of the mobile platform;

[0011] During the glue application process, the industrial control computer controls the robotic arm to move the 3D camera directly above the hole to be glued to obtain the point cloud data of the hole. The point cloud data of the hole to be glued is processed to obtain the center coordinates, radius, and depth of the hole. The amount of glue applied is calculated based on the radius, depth, and glue thickness of the hole.

[0012] The plug hopper assembly ejects a plug that matches the radius of the circular hole; after the industrial control computer controls the pneumatic gripper to remove the ejected plug, it controls the glue application assembly to move to the center coordinates and apply glue according to the specified amount. After applying glue, the industrial control computer controls the pneumatic gripper to put the removed plug into the hole, and finally controls the glue scraper to scrape off the excess glue overflowing from the surface of the hole.

[0013] Furthermore, the industrial control computer calculates the amount of adhesive applied based on the radius of the circular hole, the depth of the circular hole, and the thickness of the adhesive coating in the following manner:

[0014] The formula for the amount of adhesive applied, f, is:

[0015]

[0016] s represents the volume of adhesive remaining at the bottom of the plug after uniform application, and s is equal to the volume V of the flow channels around the plug.

[0017]

[0018] Where t is the thickness of the adhesive coating on the sidewall of the hole, r is the radius of the semi-cylinder of the guide groove, h is the hole depth, R is the radius of the circular hole, and m is the number of guide grooves on each plug, m≤4.

[0019] Furthermore, when the radius R of the circular hole is greater than 10 mm, glue is first applied to the side surface and bottom of the hole, and then a cylindrical plug with m semi-cylindrical grooves on the side is inserted into the hole using a pneumatic gripper; when the radius R of the circular hole is less than or equal to 10 mm, glue is only applied to the side surface of the hole, and then a cylindrical plug with m semi-cylindrical grooves on the side is inserted into the hole using a pneumatic gripper.

[0020] Furthermore, the industrial control computer processes the point cloud of the hole to be coated with adhesive to obtain the center coordinates, radius, and depth of the circular hole to be coated with adhesive, including:

[0021] The point cloud of the hole to be coated with adhesive is pre-processed;

[0022] Extract the depth information of the circular hole from the 3D point cloud at the bottom of the circular hole in the preprocessed point cloud;

[0023] The pre-processed point cloud of holes to be coated with adhesive is sequentially subjected to closing operation, bilateral filtering, and extraction of the edge contour of the circular holes.

[0024] The extracted circular hole contour edge is used to fit the circular hole using a circular hole fitting algorithm, and the center coordinates and radius of the fitted circle are output.

[0025] Furthermore, the step of extracting the depth information of the circular hole based on the three-dimensional point cloud at the bottom of the circular hole in the preprocessed point cloud includes:

[0026] Extract the 3D point cloud of the bottom of the circular hole in the camera coordinate system, transform the coordinates of the extracted 3D point cloud of the bottom of the circular hole to the robot arm coordinate system, and calculate the average depth value of the 3D point cloud of the bottom of the circular hole as the actual depth of the circular hole.

[0027] Furthermore, the extraction of the circular hole edge contour includes:

[0028] The image gradient is calculated on the circular hole image after image closing operation and bilateral filtering to identify the approximate outline edge of the hole. Then, the Qstu method is used to set two hysteresis thresholds, high and low, to remove false edges, thereby extracting the actual edge contour of the hole image.

[0029] Furthermore, the system also includes a force control unit installed at the end of the robotic arm; the end device also includes a hollow bracket with a pneumatic slide and a scraper plate installed on the left and right sides of the bracket, respectively, and an adhesive application assembly installed in the middle of the bracket; the pneumatic gripper assembly is installed on the pneumatic slide; and the hollow bracket is fixed to the force control unit.

[0030] Furthermore, the adhesive application assembly includes an adhesive bucket and an adhesive spray head; the automatic quantitative adhesive application and sealing system also includes a pneumatic component connected to the adhesive application assembly, and the industrial control computer controls the adhesive application assembly to perform automatic adhesive application by controlling the pneumatic component.

[0031] Furthermore, the pneumatic gripper assembly includes a sealed housing and grippers; the upper end of the sealed housing is connected to the air pump in the pneumatic assembly, and the lower end is connected to the grippers. The housing is equipped with a sensor and a drive circuit board. The drive circuit board is used to receive control commands from the industrial control computer to control the opening and closing state of the grippers. The pneumatic gripper assembly extends or retracts from the pneumatic slide under the control of the air pump.

[0032] Furthermore, the industrial control computer is also used to plan the movement path of the robotic arm; the movement path includes, in sequence, a reset point, a safety point, and a photo capture point;

[0033] The reset point is used to adjust the initial position and posture of the robotic arm after reaching a new adhesive application site; the safety point is used by the industrial control computer to control the robotic arm to reproduce the path without collision using the path information taught by the operator; after reaching the photo point, the industrial control computer controls the posture of the 3D camera at the end of the robotic arm to perform coarse positioning and obtain the image of the circular hole to be coated with adhesive through the hole number input by the host computer, and adjusts the 3D camera to perform fine positioning according to the normal vector at the center point of the plane where the circular hole to be coated with adhesive is located in the image, and obtains the image of the circular hole to be coated with adhesive again after fine positioning.

[0034] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0035] 1. The present invention provides an automatic quantitative adhesive application and sealing system based on 3D vision. It categorizes and organizes the requirements of the manual adhesive application process, and realizes the customization, flexibility and portability of the adhesive application and sealing process by constructing a complete adhesive application and sealing logic.

[0036] 2. This invention solves the problem of automatically applying adhesive to large batches of irregularly shaped circular holes with uneven distribution. The aircraft cross-section is grouped using a slicing algorithm and automatically moved to user-defined stations. Based on vision-based automatic adhesive application trajectory calculation, images of the product to be coated are acquired in real time. The industrial control computer obtains the center coordinates and radius of the circular holes to be coated through image processing, accurately locating the holes. A suitable plug is selected based on the hole radius. The amount of adhesive applied is calculated using the hole radius, hole depth, and adhesive thickness, enabling adhesive control for holes of different diameters and reducing adhesive application by 30%. Excess adhesive overflowing from the hole surface is scraped off with a scraper to ensure coating quality. This invention saves labor costs, shortens the overall production cycle, improves production efficiency, and allows for quick and convenient process changes.

[0037] 3. The industrial control computer of this invention utilizes the image processing unit to effectively process defects such as burrs and damage on the edge of the hole, reducing the number of photos taken to less than 3, and achieving a recognition success rate of over 90%.

[0038] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0039] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0040] Figure 1 This invention provides a fully automated glue application and sealing process for an automated quantitative glue application and sealing system based on 3D vision.

[0041] Figure 2 This is a schematic diagram of the plug hopper of an automatic quantitative adhesive application and sealing system based on 3D vision according to the present invention.

[0042] Figure 3 This is a schematic diagram of the end device of an automatic quantitative adhesive application and sealing system based on 3D vision according to the present invention.

[0043] Figure label:

[0044] 1-Force control;

[0045] 2-Pneumatic gripper assembly;

[0046] 3-Pneumatic slide table;

[0047] 4-Glue-coated components;

[0048] 5-Flange;

[0049] 6-3D camera;

[0050] 7-Hollow support;

[0051] 8-Glue scraper. Detailed Implementation

[0052] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0053] A specific embodiment of the present invention discloses an automated quantitative adhesive application and sealing system based on 3D vision, comprising:

[0054] Industrial control computer, mobile platform, robotic arm, 3D camera, end cap hopper assembly and end device;

[0055] The robotic arm is mounted on a movable platform, and a 3D camera and an end effector are installed at the end of the robotic arm; the industrial control computer is located inside the movable platform.

[0056] The end effector includes a pneumatic gripper assembly, an adhesive application assembly, and an adhesive scraper.

[0057] A plug hopper assembly is also placed on top of the mobile platform;

[0058] During the glue application process, the industrial control computer controls the robotic arm to move the 3D camera directly above the hole to be glued to obtain the point cloud data of the hole. The point cloud data of the hole to be glued is processed to obtain the center coordinates, radius, and depth of the hole. The amount of glue applied is calculated based on the radius, depth, and glue thickness of the hole.

[0059] The plug hopper assembly ejects a plug that matches the radius of the circular hole; after the industrial control computer controls the pneumatic gripper to remove the ejected plug, it controls the glue application assembly to move to the center coordinates and apply glue according to the specified amount. After applying glue, the industrial control computer controls the pneumatic gripper to put the removed plug into the hole, and finally controls the glue scraper to scrape off the excess glue overflowing from the surface of the hole.

[0060] Fully automated glue application and sealing process as follows Figure 1 As shown.

[0061] Specifically, the mobile platform includes an AGV chassis and a housing mounted on the chassis, with a robotic arm lifting mechanism installed inside the housing; the mobile platform is the foundation for the system to achieve full workspace coverage.

[0062] Specifically, the system also includes a host computer.

[0063] The switching between various stations can be achieved through the scheduling of the automatic cruise program in the host computer. The host computer is used to receive information about the stations and hole positions to be coated with adhesive and send it to the industrial control computer, which then controls the movable platform to move to the station to be coated with adhesive.

[0064] The host computer includes a touch screen located on the side surface of the enclosure, which is used to control the automatic navigation of the AGV and to control the program and display the interface through Qt software.

[0065] Below the display screen, there are buttons to control the automatic glue application and sealing system to be turned on or off.

[0066] Operators can control the automatic glue application and sealing system by touching the display screen and operating the buttons below, or they can remotely control the mobile platform using a remote control.

[0067] Optionally, the automatic quantitative glue application and sealing system also includes a PLC controller; the industrial computer sends the radius of the circular hole to the PLC controller, and the PLC controller controls the plug hopper assembly to eject the plug that matches the radius of the circular hole;

[0068] Preferably, the system further includes a force controller mounted at the end of the robotic arm. The force controller includes a six-dimensional force sensor.

[0069] The six-dimensional force sensor detects the simulated force and torque signals in the X, Y, and Z directions during the process of the plug entering the hole; if the force or torque in a certain direction is too large, the industrial control computer adjusts the plug's posture to achieve smooth entry into the hole.

[0070] Specifically, during the contact process with the round hole to be glued, the spray head and scraper are soft and will not damage the product to be glued. Only the plug is hard and needs to avoid bumping the product to be glued. Therefore, the force control is only activated during the plugging process, and the force control is disabled in other processes.

[0071] like Figure 3 As shown, the end device also includes a hollow bracket, with a pneumatic slide and a scraper plate installed on the left and right sides of the bracket respectively, and an adhesive application assembly installed in the middle of the bracket; the pneumatic gripper assembly is installed on the pneumatic slide; and the hollow bracket is fixed to the force control.

[0072] The adhesive application assembly includes an adhesive bucket and an adhesive spray head; the automatic quantitative adhesive application and sealing system also includes a pneumatic component, which is connected to the adhesive application assembly, and the industrial control computer controls the adhesive application assembly to perform automatic adhesive application by controlling the pneumatic component.

[0073] The pneumatic components include an air seal, an upper air pipe, a pressure valve, and an air pump.

[0074] Specifically, the air pump is located inside the movable platform; the air seal and air pressure valve are located outside the movable platform; the air pump's pipe extends from a hole on the movable platform to the outside of the platform and connects to the air pressure valve; the air pressure valve is connected to the air seal via an upper air pipe; the air seal has a through hole, the other end of which is connected to the upper end of the glue application tank; the lower end of the glue application tank is connected to the glue spray head; the industrial control computer controls the glue spray head to continuously dispense glue through the air pressure valve. The glue spray head can rotate and adjust its posture under the control of the industrial control computer.

[0075] The pneumatic gripper assembly includes a sealed housing and grippers. The upper end of the sealed housing is connected to an air pump in the pneumatic assembly, and the lower end is connected to the grippers. A sensor and a drive circuit board are installed inside the housing. The drive circuit board receives control commands from an industrial control computer to control the opening and closing state of the grippers. The pneumatic gripper assembly extends or retracts from the pneumatic slide under the control of the air pump. An indicator light is also provided on the outer surface of the sealed housing to display the self-test status of the pneumatic gripper assembly.

[0076] Specifically, when the industrial control computer receives the information that the plug has been positioned in the plug hopper assembly, it uses an air pump to control the pneumatic gripper assembly to extend from the pneumatic slide table and controls the robotic arm to move to the plug that has popped out of the plug hopper. The gripper then removes the plug. The robotic arm then moves the glue applicator assembly to directly above the hole to be glued. The pneumatic gripper assembly retracts from the pneumatic slide table. After the glue applicator completes the glue application, the pneumatic gripper assembly extends from the pneumatic slide table and inserts the plug into the hole.

[0077] Optionally, the automatic glue application and sealing system further includes a signal input device; the signal input device is used to receive control commands from the industrial control computer and send the control commands to the pneumatic gripper assembly, the glue application assembly and the glue scraper, and also to send the gripper opening and closing information of the pneumatic gripper assembly, the working status information of the air pump, the working status information of the glue scraper and the analog signal output by the force control to the industrial control computer.

[0078] Specifically, the signal input device is also connected to the pneumatic gripper assembly, the glue application assembly, the glue scraper, and the force control. After the pneumatic gripper assembly, the glue application assembly, the glue scraper, and the force control complete their self-tests, the self-test results are transmitted to the industrial control computer. The industrial control computer confirms that the self-test is correct and continues to execute the information of the next hole position or the station to be glued that was input by the user. If a problem occurs during the self-test, the host computer will issue an alarm, and the work will continue after the problem is repaired.

[0079] Optionally, after completing the self-test of the pneumatic gripper assembly, the glue application assembly, the glue scraper, and the force control, the signal input device transmits the self-test results to the host computer. The host computer confirms that the self-test is correct and continues to execute the information of the next hole or the station to be glued that was input by the user. If a problem occurs during the self-test, the host computer will issue an alarm, and the machine will continue to work after the problem is repaired.

[0080] An AGV chassis has a scanning camera at its center;

[0081] The mobile platform uses a scanning camera at the center of the chassis to scan the QR code on the ground to locate the site to be coated with adhesive. The error between the center of the mobile platform chassis and the site to be coated with adhesive is within ±2cm, which can ensure the recognition rate of the QR code when the camera is positioned.

[0082] The industrial control computer is also used to plan the movement path of the robotic arm; the movement path includes, in sequence, a reset point, a safety point, and a photo capture point.

[0083] The reset point is used to adjust the initial position and posture of the robotic arm after reaching a new adhesive application site; the safety point is used by the industrial control computer to control the robotic arm to reproduce the path without collision using the path information taught by the operator; after reaching the photo point, the industrial control computer controls the posture of the 3D camera at the end of the robotic arm to perform coarse positioning and obtain the image of the circular hole to be coated with adhesive through the hole number input by the host computer, and adjusts the 3D camera to perform fine positioning according to the normal vector at the center point of the plane where the circular hole to be coated with adhesive is located in the image, and obtains the image of the circular hole to be coated with adhesive again after fine positioning.

[0084] Upon reaching the shooting point, the industrial control computer controls the 3D camera at the end of the robotic arm to perform coarse positioning and acquire images of the circular holes to be coated with adhesive, based on the hole number input from the host computer. During coarse positioning, the normal vector of the plane containing the circular hole captured by the 3D camera must be parallel to the optical axis, which is achieved by adjusting the 3D camera's posture. During coarse positioning, the industrial control computer sends control commands to debug and set the parameters of the 3D camera, such as projection brightness, exposure times, exposure time, and white balance, in order to obtain the best shooting effect.

[0085] Then, the 3D camera position is adjusted for fine positioning based on the normal vector at the center point of the plane where the hole to be coated is located. Fine positioning is used to ensure that the normal vector at the center point of the plane where the hole is located in the image coincides with the optical axis, so that the hole in the image is located in the center of the field of view.

[0086] After precise positioning by the 3D camera, an image of the circular hole to be coated with adhesive is obtained. After image processing, the center, depth, and radius of the hole are obtained.

[0087] Preferably, in order to obtain accurate center, radius and depth of the circular hole, the present invention provides the following point cloud processing method, specifically, the method includes steps S1-S4.

[0088] Step S1: Perform preprocessing on the point cloud data of the holes to be coated, including region of interest delineation, downsampling, and point cloud filtering. Specifically, this includes steps S11-S13.

[0089] Step S11: Delineate the region of interest.

[0090] Defining the region of interest ensures that the point cloud only covers information about the circular holes, filtering out useless point clouds. Filtering out useless point clouds can improve the speed of system operation in the later stages.

[0091] First, the 3D point cloud is converted into a NumPy array. The point cloud coordinate information acquired by the 3D camera is referenced to the camera coordinate system. The region of interest (ROI) is a cylindrical area defined by the industrial control computer, with the center of the circular hole as the center, radius R, and height H. The point cloud within this ROI is used as the target point cloud for subsequent processing. The radius R and height H can be adjusted by the industrial control computer according to the actual working conditions on site.

[0092] Step S12: Downsample the point cloud within the region of interest.

[0093] The downsampling method used is voxel grid downsampling.

[0094] Point cloud downsampling reduces the density of dense point clouds to sparser ones, improving the system's operating speed in later stages. The voxel grid downsampling method divides the point cloud into multiple voxel grids, retaining only one point from the point cloud in each grid. After downsampling, the number of points in the point cloud is checked. If the number of points is less than a preset threshold, the voxel grid size is adjusted until the number of points exceeds the preset threshold; this threshold can be adjusted according to the actual on-site conditions.

[0095] Step S13: Filter the downsampled point cloud.

[0096] The point cloud filtering algorithm employs a conditional filtering algorithm. The normal vector of each point in the point cloud is calculated, and points whose normal vectors are greater than a set adjustable threshold are deleted. The remaining points are used as the target point cloud for subsequent processing.

[0097] Specifically, the normal vector of each point in the downsampled point cloud is calculated as follows:

[0098] By fitting a surface to the points around each point whose normal vector is to be calculated that are within a set radius threshold, and then calculating the normal vector of that point and the normal vectors of its neighboring points on the fitted surface, points whose normal vectors are greater than the set threshold are deleted, and the remaining points are used as the target point cloud for further processing; the set threshold can be adjusted according to the actual working conditions on site.

[0099] Step S2: Extract the depth information of the circular hole from the three-dimensional point cloud at the bottom of the circular hole in the preprocessed point cloud.

[0100] Specifically, the extraction of the hole depth based on the 3D point cloud at the bottom of the hole in the preprocessed point cloud includes:

[0101] Extract the 3D point cloud of the bottom of the circular hole in the camera coordinate system, transform the coordinates of the extracted 3D point cloud of the bottom of the circular hole to the robot arm coordinate system, and calculate the average depth value of the 3D point cloud of the bottom of the circular hole as the actual depth of the circular hole.

[0102] In the camera coordinate system, with the center of the point cloud as the origin, extract the points in the preprocessed point cloud located at the bottom of the circular hole within the radius R preset by the industrial control computer. Transform the extracted points to the robot arm coordinate system. Calculate the z-coordinate of the extracted preprocessed points at the bottom of the circular hole in the robot arm coordinate system and take the average value, which is the actual depth of the circular hole.

[0103] The origin of the robotic arm's coordinate system is the robotic arm's base, and its coordinates in the lower coordinate system are [X...]. w Y w Z w The coordinate transformation formula between the camera coordinate system and the robot arm coordinate system is as follows:

[0104]

[0105] Where R' is a 3×3 matrix representing spatial coordinate rotation; T is a 3×1 matrix representing spatial coordinate translation, [X C ,Y C Z C [ ] represents the coordinates in the camera coordinate system.

[0106] Depend on By reversing this process, the 3D coordinates of the robotic arm can be derived from the coordinates in the camera coordinate system.

[0107] Step S3: Perform closing operation, bilateral filtering, and extraction of the circular hole edge contour on the pre-processed point cloud of the hole to be coated. Specifically, this includes steps S31-S34.

[0108] S31. Convert the preprocessed point cloud of the hole to be coated to the pixel coordinate system to obtain the circular hole image.

[0109] A camera system has three coordinate systems: pixel coordinates, image coordinates, and camera coordinates. The pixel coordinate system is represented as [u,v], where the origin is the top-left corner of the image, the u-axis is horizontal to the right, and the v-axis is vertically downwards. The image coordinate system, established below the pixel coordinate system and expressed in physical units, is generally represented as (x,y), giving the pixel scale physical meaning. Its origin is the camera principal point, the intersection of the camera's optical axis and the image plane, typically located at the center of the image plane. The x-axis is parallel to the u-axis, and the y-axis is parallel to the v-axis. The camera coordinate system describes the relative position of an object to the camera and is represented as [X...]. C ,Y C Z C ], where the origin is the optical center O of the camera, X C The axis is parallel to the x-axis, and the y-axis is parallel to the x-axis. C The Z-axis is parallel to the y-axis. C The axis is parallel to the camera's optical axis and perpendicular to the image plane;

[0110] The following formula can be used to sample the coordinates [X] in the camera coordinate system. C ,Y C Z C Transform to pixel coordinates [u,v]:

[0111]

[0112] Where f' is the camera focal length, dx represents the width of one pixel in the x direction, dy represents the width of one pixel in the y direction, and (u0, v0) are the coordinates of the pixel corresponding to the center of the image plane in the pixel coordinate system.

[0113] Step S32: Perform a closing operation on the circular hole image.

[0114] The closing operation includes:

[0115] The edge contour of the hole in the circular hole image is dilated and traversed using a 3×3 convolution kernel; the edge contour of the hole in the dilated circular hole image is then eroded using the same 3×3 convolution kernel.

[0116] Closing operations refer to the sequential processing of an image through dilation and erosion. Dilation followed by erosion helps to bridge narrow gaps, eliminate small holes, and fill breaks in the contour lines.

[0117] Since the preprocessed 3D point cloud still has defects, it can significantly interfere with subsequent algorithm processing. Therefore, these defects need to be repaired to achieve the best detection results. These defects mainly manifest as discontinuities caused by inconsistent reflection at the hole edges, small voids caused by damage to the hole surface, and point cloud breaks or odd shapes caused by other data acquisition issues. To address the interference of these undesirable features on subsequent image processing and edge detection, a closing operation is used to further repair the preprocessed 3D point cloud. This closing operation first uses a 3×3 convolution kernel to dilate and traverse the hole edge contour. This process effectively repairs the broken features at the edge and eliminates small voids, making the point cloud features continuous. Then, by using the same convolution kernel to erode the continuous point cloud features, small cracks are bridged while maintaining the overall size and shape of the hole edge, resulting in a complete and continuous edge contour. These operations achieve the effect of repairing hole-sealing defects such as damage and burrs using an algorithm, effectively improving the adaptability to hole quality during automatic glue application.

[0118] Step S33: Perform bilateral filtering on the circular hole image after the closing operation; including:

[0119]

[0120]

[0121]

[0122] w = w d (i, j, k, l) × w r (i, j, k, l),

[0123] Where f(k, l) represents the grayscale value function of the corresponding pixel within the sliding window range, w d For the position domain Gaussian function, w r The grayscale Gaussian function is given by δ, where 1 ≤ k ≤ 3, 1 ≤ l ≤ 3, k and l are integers, i and j represent the x and y coordinates of the point cloud in the pixel coordinate system, respectively. d δ r The variances in the position domain and grayscale domain are represented sequentially. In a specific embodiment of the present invention, the empirical value of 0.05 obtained through experiments is used, and p(i,j) is the grayscale value of the output after bilateral filtering.

[0124] Bilateral filtering can simultaneously consider the spatial domain information and value domain information of pixels, thus it can preserve complete edges while denoising the image.

[0125] Step S34: Extract the edge contour of the circular hole from the filtered circular hole image;

[0126] Specifically, it includes:

[0127] The Canny edge detection operator is used to calculate the image gradient of the circular hole image after image closing operation and bilateral filtering to identify the approximate outline edge of the hole. Then, the Qstu method is used to set two hysteresis thresholds, high and low, to remove false edges, thereby extracting the actual edge contour of the hole image.

[0128] The calculation of image gradients includes:

[0129] By calculating the horizontal gradient information I x Vertical gradient information I y 45° gradient information I 45° and 135° gradient information I 135° The specific formula for calculating the gradient in the horizontal and vertical directions is as follows:

[0130] I x ={[p(i+1,j-1)+2p(i+1,j)+p(i+1,j+1)]-[p(i-1,j-1)+2p(i-1,j)+p(i-1,j+1)]} / 4,

[0131] I y ={[p(i-1,j+1)+2p(i,j+1)+p(i+1,j+1)]-[p(i-1,j-1)+2p(i,j-1)+p(i+1,j-1)]} / 4,

[0132] I 45° ={[2p(i+1,j+1)+p(i,j+1)+p(i+1,j)]-[2p(i-1,j-1)+p(i-1,j)+p(i,j-1)]} / 4,

[0133] I 135° ={[2p(i+1,j-1)+p(i,j-1)+p(i+1,j)]-[2p(i-1,j+1)+p(i-1,j)+p(i,j+1)]} / 4,

[0134] The formulas for calculating the gradient in the horizontal and vertical directions are:

[0135]

[0136]

[0137] By adding gradient information in both 45° and 135° directions, we can minimize the detection of a large number of false edges while losing true edge information, thus improving the accuracy of edge detection.

[0138] After the gradient calculation is completed, the approximate edge contour of the hole image can be found. However, this contour contains pseudo-edges such as isolated lines, broken lines, and bifurcation lines, which need to be removed by hysteresis thresholding to extract the actual edge contour of the hole image. The hysteresis threshold is divided into two thresholds, high and low. Points with gray values ​​above the high threshold are considered edge points; points connected to points considered as edge points are also considered edge points if they are between the high and low thresholds; points below the low threshold are considered as pseudo-edge points and are removed. Setting the high and low thresholds using the Qstu method can improve computational efficiency. This algorithm divides the image into foreground and background parts through image gray-level thresholding. The gray-level threshold for image segmentation is set to Th. The average gray level of the foreground is m1, and the ratio of pixels classified as foreground to all pixels in the image is p1; the average gray level of the background is m2, and the ratio of pixels classified as background to all pixels in the image is p2; let the global gray-level mean of the image be mG, and the inter-class variance be denoted as σ, then:

[0139] p1*m1+p2*m2=mG (1)

[0140] p1+p2=1 (2)

[0141] σ=p1(m1-mG) 2 +p2(m2-mG) 2 (3)

[0142] From equations (1)-(3), we can derive:

[0143] σ = p1 * p2(m1 - m2) 2 (4)

[0144] The inter-class variance of all pixels is calculated according to the above formula. The inter-class variance represents the degree of fluctuation between the foreground and the background. The greater the fluctuation, the more obvious the distinction between the foreground and the background. Taking the maximum value of the inter-class variance as the gray threshold Th for image segmentation can maximize the segmentation of the foreground and the background, thereby obtaining the required high threshold. The low threshold Ti is half of the high threshold. Since mG is a constant for an image, the maximum value of the inter-class variance can be obtained by transforming equation (4) into a function of m1. The maximum value of the function can be obtained by taking the derivative or finding the extreme value of the function. After obtaining the high and low thresholds, the approximate edge contour of the image can be further refined to obtain the true edge. The edge contour has eliminated redundant line segments or unclosed branches. After non-maximum suppression, non-edge points can be effectively suppressed, and the edge contour refinement is finally completed, providing a good foundation for contour fitting.

[0145] Step S4: Fit the extracted circular hole contour edge using a circular hole fitting algorithm, and output the center coordinates and radius of the fitted circle. Calculate the amount of adhesive applied based on the circular hole radius, circular hole depth, and adhesive thickness.

[0146] The steps of the circular hole fitting algorithm are as follows:

[0147] Step S41: Convert the image data in the pixel coordinate system obtained in step S3 into an M×N array K in the camera coordinate system. The element values ​​of K represent the three-dimensional coordinates of each point in the camera coordinate system.

[0148] Step S42: Calculate the average Y-axis coordinate of each column element in array K. Calculate the variance σ of the points in the extracted point cloud. If σ is greater than the preset threshold S, take another picture; otherwise, continue with the next steps.

[0149] If σ is greater than the preset threshold S, the photo is considered to be significantly skewed, and a new photo is taken.

[0150] Step S43, with The resulting 1×N array is denoted as L. Calculate the average value of array L.

[0151] Step S44: Calculate the average X-axis coordinate of each element in array K. by The resulting 1×M array is denoted as Q. Calculate the average value of array Q.

[0152] Step S45 The coordinates projected onto the robotic arm's coordinate system are the coordinates of the center point O of the fitted circle.

[0153] Step S46: Transform the points in array K located on the edge of the circular hole to the robotic arm coordinate system, calculate the Euclidean distance r' between the points in array K located on the edge of the circular hole and the center O of the fitted circle, and store the result in the array R. i Data set R i The number that accounts for the largest proportion is the radius R of the fitted circle.

[0154] The industrial control computer calculates the amount of adhesive to be applied based on the radius, depth, and thickness of the hole, applies the adhesive according to the calculated amount, and then seals the hole using a plug. Specifically, this includes:

[0155] When the radius R of the circular hole is greater than 10 mm, adhesive is first applied to the side surface and bottom of the hole, and then a cylindrical plug with m semi-cylindrical grooves on its side is inserted into the hole using a pneumatic gripper. When the radius R of the circular hole is less than or equal to 10 mm, adhesive is only applied to the side surface of the hole, and then a cylindrical plug with m semi-cylindrical grooves on its side is inserted into the hole using a pneumatic gripper. The formula for the adhesive volume f is:

[0156]

[0157] s represents the volume of adhesive remaining at the bottom of the plug after uniform application, and s is equal to the volume V of the flow channels around the plug.

[0158]

[0159] Where t is the thickness of the adhesive coating on the sidewall of the hole, r is the radius of the semi-cylinder of the guide groove, h is the hole depth, and m≤4.

[0160] The automatic quantitative adhesive application and sealing system uses a spray nozzle that is finely adjusted according to the center coordinates of the circular hole in the robotic arm's base coordinate system, and applies adhesive according to the set amount of adhesive.

[0161] An infrared sensor is installed on the top of the plug hopper assembly to detect whether the plug is in place or has been removed; the output signal of the infrared sensor is transmitted to the industrial control computer.

[0162] The plug hopper assembly includes a lifting mechanism and a plug hopper; a schematic diagram of the plug hopper is shown below. Figure 2 As shown.

[0163] The plug hopper is placed on a lifting mechanism with a translation function. When the plug hopper assembly needs to be discharged, the lifting mechanism moves to directly below the plug of the specified size and lifts the required plug. After receiving the plug arrival information sent by the infrared sensor, the industrial control computer controls the gripper to pick up the plug. After applying glue, the plug is placed into the hole. After detecting the material picking completion information sent by the infrared sensor, the industrial control computer controls the lifting mechanism to retract and wait for the next discharge command.

[0164] Optionally, the output signal of the infrared sensor is transmitted to the PLC controller. After receiving the plug positioning information sent by the infrared sensor, the PLC controller sends a command to the industrial computer, which controls the gripper to remove the plug. After applying adhesive, the plug is placed into the hole. After detecting the material handling completion information sent by the infrared sensor, the PLC controller controls the lifting mechanism to retract, waiting for the next material discharge command.

[0165] Specifically, the lifting mechanism is controlled by a linear motor. The PLC controller is also used to send the status of the plug hopper assembly to the host computer. When it detects the material handling completion information sent by the infrared sensor, it sends the material handling completion information to the host computer. After the plug hopper assembly executes each instruction, the PLC controller performs a self-check on the status of the plug hopper assembly and sends the check result to the host computer. The host computer judges the self-check status of the plug hopper assembly, pneumatic gripper assembly, glue application assembly, glue scraper, and force control. If the status is normal, it continues to execute the task. After the task is completed, it receives the information of the new glue application site and hole position input by the user.

[0166] Compared with existing technologies, the 3D vision-based automatic quantitative adhesive application and sealing system provided in this embodiment categorizes and organizes the requirements of the manual adhesive application process, and achieves customization, flexibility, and portability of the adhesive application and sealing process by constructing a complete adhesive application and sealing logic. This invention solves the problem of automatic adhesive application to large batches of irregularly shaped circular holes with uneven distribution. The aircraft cross-section is grouped using a slicing algorithm and automatically moved to user-defined stations. Based on vision, the automatic adhesive application trajectory is calculated, and images of the product to be coated are acquired in real time. The industrial control computer obtains the center coordinates and radius of the circular holes to be coated through image processing, accurately locates the position of the holes, selects appropriate plugs based on the hole radius, and calculates the adhesive amount based on the hole radius, hole depth, and adhesive thickness. This achieves adhesive amount control for holes of different diameters, reducing the adhesive amount by 30%. Excess adhesive overflowing from the hole surface is scraped off by a scraper to ensure adhesive quality. This system saves labor costs, shortens the overall production cycle, improves production efficiency, and allows for quick and convenient process changes. The industrial control computer of this invention utilizes an image processing unit to effectively process defects such as burrs and damage on the edge of holes, reducing the number of photos taken to less than 3, and achieving a recognition success rate of over 90%.

[0167] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by an industrial control computer program instructing related hardware, and the program can be stored in an industrial control computer-readable storage medium. The industrial control computer-readable storage medium can be a disk, optical disk, read-only memory, or random access memory, etc.

[0168] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An automated quantitative adhesive application and sealing system based on 3D vision, characterized in that, include: Industrial control computer, mobile platform, robotic arm, 3D camera, end cap hopper assembly and end device; The robotic arm is mounted on a movable platform, and a 3D camera and an end effector are installed at the end of the robotic arm; the industrial control computer is located inside the movable platform. The end effector includes a pneumatic gripper assembly, an adhesive application assembly, and an adhesive scraper. A plug hopper assembly is also placed on top of the mobile platform; During the glue application process, the industrial control computer controls the robotic arm to move the 3D camera directly above the hole to be glued to obtain the point cloud data of the hole. The point cloud data of the hole to be glued is processed to obtain the center coordinates, radius, and depth of the hole. The amount of glue applied is calculated based on the radius, depth, and glue thickness of the hole. The industrial computer calculates the amount of adhesive applied based on the radius of the circular hole, the depth of the circular hole, and the thickness of the adhesive coating, using the following method: The formula for the amount of adhesive applied, f, is: s represents the volume of adhesive remaining at the bottom of the plug after uniform application, and s is equal to the volume V of the flow channels around the plug. ， Where t is the thickness of the adhesive coating on the sidewall of the hole, r is the radius of the semi-cylinder of the guide groove, h is the hole depth, R is the radius of the circular hole, m is the number of guide grooves on each plug, and m≤4; When the radius R of the circular hole is greater than 10 mm, apply glue to the side surface and bottom of the hole first, and then use a pneumatic gripper to put a cylindrical plug with m semi-cylindrical grooves on the side into the hole; when the radius R of the circular hole is less than or equal to 10 mm, apply glue only to the side surface of the hole, and then use a pneumatic gripper to put a cylindrical plug with m semi-cylindrical grooves on the side into the hole. The plug hopper assembly ejects a plug that matches the radius of the circular hole; after the industrial control computer controls the pneumatic gripper to remove the ejected plug, it controls the glue application assembly to move to the center coordinates and apply glue according to the specified amount. After applying glue, the industrial control computer controls the pneumatic gripper to put the removed plug into the hole, and finally controls the glue scraper to scrape off the excess glue overflowing from the surface of the hole.

2. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 1, characterized in that, The industrial control computer processes the point cloud of the hole to be coated with adhesive to obtain the center coordinates, radius, and depth of the circular hole, including: The point cloud of the hole to be coated with adhesive is pre-processed; Extract the depth information of the circular hole from the 3D point cloud at the bottom of the circular hole in the preprocessed point cloud; The pre-processed point cloud of holes to be coated with adhesive is sequentially subjected to closing operation, bilateral filtering, and extraction of the edge contour of the circular holes. The circular hole is fitted using a circular hole fitting algorithm based on the extracted circular hole contour edge, and the center coordinates and radius of the fitted circle are output.

3. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 2, characterized in that, The extraction of hole depth information based on the 3D point cloud at the bottom of the hole in the preprocessed point cloud includes: Extract the 3D point cloud of the bottom of the circular hole in the camera coordinate system, transform the coordinates of the extracted 3D point cloud of the bottom of the circular hole to the robot arm coordinate system, and calculate the average depth value of the 3D point cloud of the bottom of the circular hole as the actual depth of the circular hole.

4. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 2, characterized in that, The extraction of the circular hole edge contour includes: The image gradient is calculated on the circular hole image after image closing operation and bilateral filtering to identify the approximate outline edge of the hole. Then, the Qstu method is used to set two hysteresis thresholds, high and low, to remove false edges, thereby extracting the actual edge contour of the hole image.

5. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 1, characterized in that, The system also includes a force controller, which is installed at the end of the robotic arm; the end device also includes a hollow bracket, on the left and right sides of which a pneumatic slide and a scraper are respectively installed, and an adhesive applicator is installed in the middle of the bracket; the pneumatic gripper assembly is installed on the pneumatic slide; the hollow bracket is fixed to the force controller.

6. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 5, characterized in that, The adhesive application assembly includes an adhesive bucket and an adhesive spray head; the automatic quantitative adhesive application and sealing system also includes a pneumatic component, which is connected to the adhesive application assembly, and the industrial control computer controls the adhesive application assembly to perform automatic adhesive application by controlling the pneumatic component.

7. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 5, characterized in that, The pneumatic gripper assembly includes a sealed housing and grippers; the upper end of the sealed housing is connected to the air pump in the pneumatic assembly, and the lower end is connected to the grippers. The housing is equipped with a sensor and a drive circuit board. The drive circuit board is used to receive control commands from the industrial control computer to control the opening and closing state of the grippers; the pneumatic gripper assembly extends or retracts from the pneumatic slide under the control of the air pump.

8. The automatic quantitative adhesive application and sealing system based on 3D vision according to claim 1, characterized in that, The industrial control computer is also used to plan the movement path of the robotic arm; the movement path includes, in sequence, a reset point, a safety point, and a photo capture point. The reset point is used to adjust the initial position and posture of the robotic arm after reaching a new adhesive application site; the safety point is used by the industrial control computer to control the robotic arm to reproduce the path without collision using the path information taught by the operator; after reaching the photo point, the industrial control computer controls the posture of the 3D camera at the end of the robotic arm to perform coarse positioning and obtain the image of the circular hole to be coated with adhesive through the hole number input by the host computer, and adjusts the 3D camera to perform fine positioning according to the normal vector at the center point of the plane where the circular hole to be coated with adhesive is located in the image, and obtains the image of the circular hole to be coated with adhesive again after fine positioning.

Images