Visual inspection method and visual inspection system for gluing breakpoints

Abstract

The invention provides a visual detection method and a visual detection system for gluing breakpoints. The visual detection method comprises the following steps: determining a position relationship between a glue gun of a gluing robot and an image acquisition device; a gluing robot is controlled to drive a glue gun to glue along a preset track, and a continuous breakpoint-free standard gluing track is obtained; while the glue gun performs gluing along the preset track, standard gluing images of the standard gluing track are collected at preset shooting intervals through an image collecting device; obtaining gluing information in the processed standard gluing image through deep learning; in the actual working process that the gluing robot conducts gluing along the preset track, working gluing images are collected at preset shooting intervals through an image collecting device; and based on the working gluing image and the gluing information, determining a breakpoint occurrence condition and a breakpoint position in the working gluing track. According to the visual detection method and the visual detection system provided by the invention, the judgment and positioning of the gluing breakpoint can be automatically completed, and the deviation detection capability and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the technical field of automobile gluing, and in particular to a visual detection method and a visual detection system for gluing breakpoints. Background Art

[0002] In the production of automobiles, there are many gluing stations. The gluing industry is needed in the manufacturing process of engines, windows, doors and other parts. During the gluing process, the gluing robot sometimes breaks and leaks glue, resulting in breakpoints in the gluing track and causing gluing quality defects.

[0003] In order to ensure the quality of glue coating, glue coating detection has become an indispensable link. Currently, there is a lack of means to locate glue breakage, and it can only be judged manually, and the ability and accuracy of detecting deviations are low. Summary of the invention

[0004] To at least partially solve the above-mentioned problems, according to the first aspect of the present application, an embodiment of the present application provides a visual detection method for gluing breakpoints, the method is used to detect whether a breakpoint occurs in a gluing track formed by glue when a glue gun of a gluing robot is gluing, and the position of the breakpoint, the method comprising: determining the positional relationship between the glue gun of the gluing robot and an image acquisition device, the image acquisition device being fixed on the gluing robot, and the image acquisition device being used to acquire the gluing track; controlling the gluing robot to drive the glue gun to apply glue along a preset track to obtain a standard gluing track that is continuous and has no breakpoints; while the glue gun is applying glue along the preset track, a standard gluing image of the standard gluing track is acquired by the image acquisition device at a preset shooting interval; the standard gluing image is processed, and then the gluing information in the processed standard gluing image is obtained by deep learning; in the actual working process of the gluing robot applying glue along the preset track, a working gluing image of the working gluing track formed by spraying of the glue gun is acquired by the image acquisition device at a preset shooting interval; based on the working gluing image and the gluing information, the occurrence of breakpoints and the position of breakpoints in the working gluing track are determined.

[0005] In some embodiments, the standard glue-coated image is processed, including: stitching multiple standard glue-coated images into a standard complete glue-coated image containing a complete standard glue-coated trajectory; performing image enhancement, filtering and denoising, binarization, morphological processing, centerline extraction and edge detection on the standard complete glue-coated image to obtain a processed standard glue-coated image.

[0006] In some embodiments, the gluing information in the processed standard gluing image is obtained through deep learning, including: obtaining multiple skeleton points along the extension direction of the standard gluing track, each skeleton point is the midpoint of the corresponding standard gluing track in the width direction; the position information of each skeleton point and the width information of the standard gluing track at the skeleton point are recorded in a set of vectors to obtain the gluing information.

[0007] In some embodiments, based on the working gluing image and the gluing information, the occurrence and positions of breakpoints in the working gluing trajectory are determined, including: performing the same processing on the working gluing image as the standard gluing image to obtain a processed working gluing image; fitting the complete working gluing trajectory contained in the processed working gluing image with the complete standard gluing trajectory; based on the fitting results, determining the positions of multiple skeleton points on the complete working gluing trajectory; based on the positions of multiple skeleton points on the complete working gluing trajectory and the width information corresponding to the skeleton points, determining the occurrence and positions of breakpoints in the working gluing trajectory.

[0008] In some embodiments, based on the positions of multiple skeleton points on the complete working gluing trajectory and the width information corresponding to the skeleton points, the occurrence and positions of breakpoints in the working gluing trajectory are determined, including: on the complete working gluing trajectory, for each skeleton point, obtaining the target number of pixels at the skeleton point along the width direction of the complete working gluing trajectory; in response to the target number of pixels being less than a preset threshold, determining that there is a breakpoint in the working gluing trajectory, the preset threshold being determined based on the width information; and determining the breakpoint position based on the position of the skeleton point where the target number of pixels is less than the preset threshold.

[0009] In some embodiments, after determining the breakpoint position, it also includes: controlling the gluing robot to add glue to the breakpoint position.

[0010] In some embodiments, the gluing information also includes direction information at each skeleton point, and controlling the gluing robot to fill the breakpoint position with glue includes: generating a gluing trajectory based on the position information and direction information of the skeleton point; and controlling the gluing robot to fill the breakpoint position with glue along the gluing trajectory.

[0011] In some embodiments, the glue filling trajectory is different from the preset trajectory.

[0012] In some embodiments, before glue repair, the glue gun of the glue coating robot is controlled to be in a standby position, and the standby position is located outside a preset trajectory; the glue repair trajectory is a trajectory from the standby position to the breakpoint position and back to the standby position; when there are multiple breakpoint positions, multiple trajectories are generated from the standby position to the breakpoint position and back to the standby position.

[0013] According to the second aspect of the present application, an embodiment of the present application further provides a visual inspection system for gluing breakpoints, and the visual inspection system is used to execute the method provided in any embodiment of the first aspect of the present application.

[0014] The visual detection method and visual detection system for gluing breakpoints provided in the embodiments of the present application collect standard gluing images through an image acquisition device, and obtain gluing information through image processing and deep learning, and then determine the occurrence and position of breakpoints in the working gluing trajectory based on the working gluing images collected during the actual working process. Therefore, the visual detection method and visual detection system for gluing breakpoints provided in the embodiments of the present application can automatically complete the judgment and positioning of gluing breakpoints, and improve the ability and accuracy of detecting deviations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 is a flow chart of a visual detection method for glue coating breakpoints provided in an embodiment of the present application;

[0017] Figure 2 It is a schematic diagram of a processed standard glued image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical scheme of the preferred embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection claimed in the present application.

[0019] According to the first aspect of the present application, an embodiment of the present application provides a visual detection method for gluing breakpoints, the detection method is used to detect whether a breakpoint and the position of the breakpoint appear in the gluing track formed by glue when the glue gun of the gluing robot is gluing, and the method includes: determining the positional relationship between the glue gun of the gluing robot and an image acquisition device, the image acquisition device is fixed on the gluing robot, the image acquisition device can be a camera, and the image acquisition device is used to acquire the gluing track; controlling the gluing robot to drive the glue gun to glue along a preset track to obtain a continuous standard gluing track without breakpoints; while the glue gun is gluing along the preset track, a standard gluing image of the standard gluing track is acquired by the image acquisition device at a preset shooting interval; the standard gluing image is processed, and then the gluing information in the processed standard gluing image is obtained by deep learning; in the actual working process of the gluing robot gluing along the preset track, a working gluing image of the working gluing track formed by spraying of the glue gun is acquired by the image acquisition device at a preset shooting interval; based on the working gluing image and the gluing information, the occurrence of breakpoints and the position of breakpoints in the working gluing track are determined.

[0020] Specifically, combined Figure 1 In actual production, the glue gun is fixed on the mechanical arm of the industrial glue robot. First, the equipment parameters are set, the camera calibration is performed, and the hand-eye calibration of the glue robot is realized. When the positional relationship between the glue gun of the glue robot and the image acquisition device is determined, the relationship between the camera coordinate system and the flange coordinate system at the end of the robot's mechanical arm can be established. After the camera is calibrated, the standard glue image is collected and image processing is performed on the standard glue image. Then, the processed standard glue image is used for deep learning to obtain the glue information in the standard glue image. The glue information can include the number information of the glue track skeleton points, the position information of the glue track skeleton points, the glue track width information at the glue track skeleton points, the direction information at the glue track skeleton points, etc. The glue information can be stored in multiple sets of vectors for subsequent judgment and positioning of glue breakpoints. In the actual working process of the robot, the working glue image of the working glue track is collected in real time and image processing is performed to obtain the glue information in the working glue image. Then, the glue information obtained through training is accelerated to determine whether there is a breakpoint and the position of the breakpoint.

[0021] The visual detection method and visual detection system for gluing breakpoints provided in the embodiments of the present application collect standard gluing images through an image acquisition device, and obtain gluing information through image processing and deep learning, and then determine the occurrence and position of breakpoints in the working gluing trajectory based on the working gluing images collected during the actual working process. Therefore, the visual detection method and visual detection system for gluing breakpoints provided in the embodiments of the present application can automatically complete the judgment and positioning of gluing breakpoints, and improve the ability and accuracy of detecting deviations.

[0022] In some embodiments, the standard glue coating image is processed, including: stitching together multiple standard glue coating images into a standard complete glue coating image that includes a complete standard glue coating track; performing image enhancement, filtering and denoising, binarization, morphological processing, centerline extraction, and edge detection on the standard complete glue coating image to obtain a processed standard glue coating image. It can be understood that as the image acquisition device moves with the glue gun, the glue gun applies glue in real time, and the image acquisition device cannot capture the complete glue coating track image at one time. Therefore, in this embodiment, a complete glue coating track image, i.e., a standard complete glue coating image, is obtained by stitching together multiple images. The processed partial standard glue coating image is shown in FIG. Figure 2 As shown in the figure, after image processing, the factors in the image that have little correlation with the gluing trajectory can be removed, and the characteristics of the gluing trajectory can be highlighted, which is convenient for subsequent deep learning.

[0023] In some embodiments, the gluing information in the processed standard gluing image is obtained through deep learning, including: obtaining multiple skeleton points along the extension direction of the standard gluing track, each skeleton point is the midpoint of the width direction of the standard gluing track to which it corresponds; recording the position information of each skeleton point and the width information of the standard gluing track at the skeleton point in a set of vectors to obtain the gluing information. In this embodiment, recording the characteristics of the standard gluing track through skeleton points can simplify data processing and improve the calculation speed when determining and locating breakpoints in subsequent processes. Along the extension direction of the standard gluing track, skeleton points can be obtained at preset distances. The smaller the preset distance, the more accurately the skeleton points can depict the standard gluing track information.

[0024] In some embodiments, based on the working glue coating image and the glue coating information, the occurrence and position of breakpoints in the working glue coating trajectory are determined, including: performing the same processing on the working glue coating image as the standard glue coating image to obtain the processed working glue coating image; fitting the complete working glue coating trajectory contained in the processed working glue coating image with the complete standard glue coating trajectory; based on the fitting result, determining the positions of multiple skeleton points on the complete working glue coating trajectory; based on the positions of multiple skeleton points on the complete working glue coating trajectory and the width information corresponding to the skeleton points, determining the occurrence and position of breakpoints in the working glue coating trajectory. In this embodiment, the processing method for the working glue coating image can also be to stitch multiple working glue coating images into a working complete glue coating image containing a complete working glue coating trajectory; performing image enhancement, filtering and denoising, binarization, morphological processing, centerline extraction and edge detection on the working complete glue coating image to obtain the processed working glue coating image. By using the same processing method and processing parameters, it can be ensured that the glue coating information in the working glue coating image and the standard glue coating image corresponds to each other, which is convenient for comparison. The complete working gluing trajectory contained in the processed working gluing image is then fitted with the complete standard gluing trajectory to determine the correspondence between the complete working gluing trajectory and the complete standard gluing trajectory. Furthermore, based on the position of the skeleton point in the complete standard gluing trajectory, the position of the corresponding skeleton point in the complete working gluing trajectory is determined. Then, the width information corresponding to the skeleton point is compared to determine the occurrence and position of breakpoints in the working gluing trajectory.

[0025] In some embodiments, based on the positions of multiple skeleton points on the complete working gluing track and the width information corresponding to the skeleton points, the occurrence and position of breakpoints in the working gluing track are determined, including: on the complete working gluing track, for each skeleton point, the number of target pixels along the width direction of the complete working gluing track at the skeleton point is obtained; in response to the number of target pixels being less than a preset threshold, it is determined that there is a breakpoint in the working gluing track, and the preset threshold is determined based on the width information; based on the position of the skeleton point where the number of target pixels is less than the preset threshold, the position of the breakpoint is determined. In this embodiment, with the skeleton point as the center, an up and down search is performed along the width direction of the complete working gluing track at the skeleton point, and the search direction and search width are reasonably set to determine the number of target pixels within the search range. If the number of target pixels within the search range is greater than or equal to the preset threshold, it can be determined that there is glue at the position of the skeleton point. If the number of target pixels within the search range is less than the preset threshold, it can be determined that there is no glue or glue is missing at the position of the skeleton point, and a breakpoint occurs. The position of the breakpoint is determined according to the position of the skeleton point, thereby achieving the acquisition of the position of the breakpoint and the quality of glue within the shooting range. Finally, the comparison information is fed back to the host computer in real time to obtain the precise starting and ending coordinates of the breakpoint.

[0026] In some embodiments, after determining the breakpoint position, the process further includes: controlling the glue coating robot to fill the breakpoint position with glue. In this embodiment, by setting the glue filling operation, the whole process of glue coating quality monitoring, breakpoint position determination and automatic glue filling is realized, further improving the operation efficiency and automation.

[0027] In some embodiments, the gluing information also includes direction information at each skeleton point, and controlling the gluing robot to fill the breakpoint position includes: generating a gluing track based on the position information and direction information of the skeleton point; and controlling the gluing robot to fill the breakpoint position along the gluing track. In this embodiment, the position of the gluing breakpoint can be determined and the path information to the gluing breakpoint position can be generated through the position information and direction information of the skeleton point, so as to realize automatic gluing.

[0028] In some embodiments, the glue filling track is different from the preset track. In this embodiment, when filling glue, the glue gun does not move along the preset track to prevent the glue gun from damaging the area where glue has been applied when filling glue.

[0029] In some embodiments, before glue repair, the glue gun of the glue coating robot is controlled to be in a standby position, and the standby position is located outside the preset trajectory; the glue repair trajectory is a trajectory from the standby position to the breakpoint position and back to the standby position; when there are multiple breakpoint positions, multiple trajectories are generated from the standby position to the breakpoint position and back to the standby position. In this embodiment, when the glue gun is repairing glue, it starts from the standby position to repair glue to multiple breakpoints in sequence, returns to the standby position after completing the repair of a single breakpoint, and then starts again to reach the next breakpoint position to repair glue, thereby preventing damage to the area where glue has been applied.

[0030] According to the second aspect of the present application, the embodiment of the present application also provides a visual detection system for gluing breakpoints, and the visual detection system is used to perform the method provided by any embodiment of the first aspect of the present application. The visual detection system may include a glue gun, an image acquisition device (such as an industrial camera, etc.), a host computer, an industrial computer, an industrial robot (gluing robot), and the camera and the glue gun are fixed on the industrial robot. When executing the visual detection method, the industrial robot first performs hand-eye calibration, establishes the relationship between the camera coordinate system and the robot end flange coordinate system, the robot establishes a workpiece coordinate system, and designs multiple marking points on the robot's gluing trajectory. When the industrial robot drives the camera to obtain continuous acquisition images, the gluing position and the glue breaking position in the image are detected by image processing, and the position of the marking point on the robot trajectory, the camera's shooting time, and the pixel position of the breakpoint in the image are combined with the camera's hand-eye relationship to calculate the starting coordinates and end coordinates of the gluing breakpoint in the workpiece coordinate system. The robot's online visual detection system for gluing quality with a positioning function can guide the robot to perform precise position glue filling. The present invention can automatically compare and judge the real-time glue coating image with the glue coating information obtained through algorithm training, rather than manually completing the empirical judgment task, greatly improving the ability and accuracy of detecting deviations. The present invention can accurately obtain the starting coordinates and end coordinates of the breakpoints in the glue coating process, and has the characteristics of real-time high-precision online detection of machine vision, which can improve the efficiency of quality inspection while ensuring the reliability of the inspection results.

[0031] Based on the above-mentioned embodiments of the present application, in the absence of explicit negation or conflict, the technical features of one embodiment may be beneficially combined with one or more other embodiments.

[0032] Although some specific embodiments of the present application have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are only for illustration and not for limiting the scope of the present application. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A visual detection method for glue breakpoints, the method is used to detect whether a breakpoint appears in the glue track formed by glue when the glue gun of the glue robot is applying glue and the position of the breakpoint, characterized in that: The method comprises: Determine the positional relationship between the glue gun of the glue coating robot and an image acquisition device, wherein the image acquisition device is fixed on the glue coating robot and is used to acquire the glue coating trajectory; Control the glue coating robot to drive the glue gun to apply glue along a preset trajectory to obtain a continuous standard glue coating trajectory without breakpoints; While the glue gun is applying glue along the preset track, the image acquisition device is used to acquire a standard glue application image of the standard glue application track at a preset shooting interval; Processing the standard glue-coated image, and then obtaining glue-coating information in the processed standard glue-coated image through deep learning; During the actual working process of the gluing robot applying glue along the preset track, the image acquisition device acquires the working gluing image of the working gluing track formed by the glue gun spraying at the preset shooting interval; Based on the working gluing image and the gluing information, the occurrence and position of breakpoints in the working gluing track are determined.

2. The method according to claim 1, characterized in that The standard glued image is processed, including: splicing a plurality of the standard glue-coated images into a standard complete glue-coated image including a complete standard glue-coated track; The standard complete glue-coated image is subjected to image enhancement, filtering and denoising, binarization, morphological processing, centerline extraction and edge detection to obtain a processed standard glue-coated image.

3. The method according to claim 2, characterized in that The glue information in the processed standard glue image is obtained through deep learning, including: Acquire a plurality of skeleton points along the extension direction of the standard gluing track, each of the skeleton points being the midpoint of the corresponding standard gluing track in the width direction; The position information of each skeleton point and the width information of the standard gluing track at the skeleton point are recorded in a set of vectors to obtain the gluing information.

4. The method according to claim 3, characterized in that Based on the working gluing image and the gluing information, determining the occurrence and position of breakpoints in the working gluing track includes: Performing the same processing on the working glued image as on the standard glued image to obtain a processed working glued image; Fitting the complete working gluing trajectory contained in the processed working gluing image with the complete standard gluing trajectory; Based on the fitting result, determining the positions of the plurality of skeleton points on the complete working gluing trajectory; Based on the positions of the plurality of skeleton points on the complete working gluing track and the width information corresponding to the skeleton points, the occurrence and positions of breakpoints in the working gluing track are determined.

5. The method according to claim 4, characterized in that Based on the positions of the plurality of skeleton points on the complete working gluing track and the width information corresponding to the skeleton points, determining the occurrence and positions of breakpoints in the working gluing track, including: On the complete working gluing track, for each of the skeleton points, obtaining the target number of pixels at the skeleton point along the width direction of the complete working gluing track; In response to the target pixel number being less than a preset threshold, determining that a breakpoint exists in the working gluing track, the preset threshold being determined based on the width information; The breakpoint position is determined based on the position of the skeleton point where the number of target pixels is less than a preset threshold.

6. The method according to claim 5, characterized in that After the breakpoint position is determined, the method further includes: The glue coating robot is controlled to fill glue at the breakpoint position.

7. The method according to claim 6, characterized in that The gluing information also includes direction information at each skeleton point, and controlling the gluing robot to fill the breakpoint position with glue includes: Generate a glue filling trajectory based on the position information of the skeleton point and the direction information; The glue coating robot is controlled to fill glue at the breakpoint position along the glue filling trajectory.

8. The method according to claim 7, characterized in that The glue filling trajectory is different from the preset trajectory.

9. The method according to claim 8, characterized in that Before glue filling, the glue gun of the glue coating robot is controlled to be in a standby position, and the standby position is outside the preset track; The glue filling trajectory is a trajectory from the standby position to the breakpoint position and then back to the standby position; When there are multiple breakpoint positions, multiple trajectories are generated, which respectively reach the breakpoint positions from the waiting position and return to the waiting position.

10. A visual inspection system for glue breakpoints, characterized in that: The visual inspection system is used to perform the method according to any one of claims 1-9.

Images