Detection method, system and equipment for gluing process and medium

Through deep learning technology, the glue strip area is positioned and extracted, combined with the SVD method to orderly point set and normal direction search, the problem of difficulty in taking into account efficiency and accuracy in the existing glue coating detection technology is solved, and the glue coating process is realized Efficient and accurate detection.

CN119027407APending Publication Date: 2024-11-26ANHUI JEE AUTOMATION EQUIP CO LTD

Patent Information

Application Number
CN202411199032.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing glue coating detection technology is difficult to take into account the detection efficiency and accuracy, especially when the glue coating process changes, it requires a large number of manual adjustments, and deep learning technology is prone to interference and misjudgment during glue width detection.

Method used

The image frame is obtained by using a convolutional neural network based on deep learning, positioning and extracting the rubber strip area, determining the starting point and obtaining the trajectory detection point set, orderly adjusting the point set and calculating the rubber strip length through the SVD method to determine whether the rubber strip is abnormal. , use the RexNet50 network and RPN structure to classify the foreground and background of the rubber strip, correct the anchor frame position, search the edge of the rubber strip along the normal direction, and determine whether the rubber strip width exceeds the preset range.

Benefits of technology

It improves the accuracy and efficiency of glue coating detection, adapts to changes in glue coating process, reduces the need for manual adjustment of parameters, and enhances the accuracy and robustness of glue strip width detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027407A_ABST
    Figure CN119027407A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of process detection, and provides a gluing process detection method, system and device and a medium, and the method comprises the steps: obtaining a workpiece image frame in a gluing process in real time, and positioning and extracting an adhesive tape region through a deep learning-based convolutional neural network; determining an adhesive tape starting point according to the number of the adhesive tape areas, and then performing track detection on the adhesive tape areas to obtain an adhesive tape track detection point set; ordering the track detection point set, and calculating the length of the adhesive tape at the pixel point; and finally, judging whether the gluing area is abnormal or not according to the adhesive tape area and the pixel point length. According to the method, the accuracy and efficiency of gluing detection can be effectively improved, and the change requirements of a gluing process can be better met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of process detection, and in particular to a detection method, system, equipment and medium for a glue coating process. Background Art

[0002] With the refinement and automation of the glue coating process, automatic optical inspection technology is widely used in the glue coating field.

[0003] At present, in the traditional 2D visual gluing inspection technology, gluing inspection is usually carried out according to a preset template. However, the preset template cannot adapt to the changes in the on-site gluing process, and professionals need to spend a lot of time and energy to adjust the parameters.

[0004] For example, existing 2D vision technology can realize the function of automatic glue detection. The patent "A method and device for detecting glue coating of diesel engines based on 2D machine vision", application number CN202311694889.3, uses the background difference method to extract the glue coating area, which has high requirements for the consistency of the image acquisition environment, and manually presets multiple capture frames for glue path detection, which has low detection efficiency.

[0005] In the 2D visual gluing inspection technology based on deep learning, it has high robustness in locating gluing defects, but it is difficult to obtain accurate data such as colloid width for final judgment of gluing inspection results.

[0006] For example: The patent "Glue coating detection method, system, device and storage medium", application number CN202311184004.5, uses deep learning to extract the glue path area, and the skeleton smoothing algorithm determines that the center line of the glue path is easily disturbed by bifurcation, and the vertical interval detection line method used to judge the width in the glue width detection stage is special and easy to cause misjudgment.

[0007] In summary, the prior art cannot well balance detection efficiency and detection accuracy when performing abnormality detection in the glue coating process. Summary of the invention

[0008] To solve the above problems, the present invention discloses a method, system, equipment and medium for detecting a glue coating process, which adopts the method of acquiring image frames, locating and extracting the glue strip area using a convolutional neural network, determining the starting point and acquiring a trajectory detection point set, ordering the point set and calculating the glue strip length, and judging abnormalities based on the glue strip area and pixel length to realize the method of detecting the glue coating area, which can improve the accuracy and efficiency of glue coating detection and adapt to changes in the glue coating process.

[0009] Real-time acquisition of workpiece image frames during the gluing process;

[0010] The convolutional neural network based on deep learning is used to locate and extract the rubber strip area in the workpiece image frame to obtain the rubber strip area;

[0011] Determine the starting point of the rubber strip, perform trajectory detection on the rubber strip area through the starting point, and obtain the rubber strip trajectory detection point set

[0012] The discrete trajectory detection point set Ordered so that each pixel C i The nearest neighbor point f i and b i are the two points in front and behind along the length direction of the rubber strip in the actual position; according to each pixel point C i And the two nearest points are obtained by SVD method i The normal direction of the position n i ; Along the normal direction n i The positive and negative directions are searched simultaneously until the edge of the rubber strip is found. The rubber strip is at pixel point C. i The length is L i =a i +b i , where a i , b i is the length of the rubber strip along the positive and negative semi-axis in the normal direction;

[0013] According to the strip area and pixel C i The length of the glue coating area can be used to determine whether it is abnormal.

[0014] Further,

[0015] The convolutional neural network based on deep learning is used to locate and extract the rubber strip area in the workpiece image frame to obtain the rubber strip area;

[0016] include:

[0017] Send the image to the convolutional layer to obtain the feature map of the rubber strip, and the convolutional layer is the convolution module of the RexNet50 network;

[0018] The extracted rubber strip feature map is sent to the RPN structure to classify the rubber strip foreground and background into two categories, and the anchor frame information of the rubber strip foreground area is obtained;

[0019] The regression module is used to correct the position of the anchor frame in the rubber strip foreground area and output the position of the rubber strip area.

[0020] Further,

[0021] Determining the starting point of the rubber strip comprises:

[0022] If the number of glue strip areas is 0, it means that there is glue breakage in the current image frame, and the determination of the glue strip starting point is stopped at this time.

[0023] Further,

[0024] Determining the starting point of the rubber strip comprises:

[0025] If the number of glue strip areas is 1, the glue nozzle position of the glue coating device is set to the glue strip starting position point L0.

[0026] Further,

[0027] Determining the starting point of the rubber strip comprises:

[0028] If the number of rubber strip regions is not less than 2, the distance transformation algorithm is used to find the maximum distance of each rubber strip region as the starting position point L0 of the rubber strip;

[0029] L0=Max(D(p))=Max(Min(disf(p,q)))p∈O,q∈B

[0030] Among them, O is the set of foreground glue road areas, B is the set of background areas, and the distance map is D.

[0031] Further,

[0032] The rubber strip area is tracked by the starting point to obtain a rubber strip track detection point set. include:

[0033] Generate a circular area R0′ at L0, the center of R0′ coincides with L0, and the radius is the ideal glue width;

[0034] Perform threshold segmentation on the R0′ region to separate the rubber strip foreground from the background image, find the centroid C0 of the rubber strip foreground region, generate a circular region R0 of the same size as R0′ at C0, and determine the direction of the rubber strip trajectory generation based on the characteristic that the length of the rubber strip is larger than its width;

[0035] Generate a second circular area along the direction of the rubber strip track at a distance equal to the preset interval point distance. Repeat the operation to obtain a series of rubber strip track detection point sets.

[0036] Further,

[0037] According to the strip area and pixel point C i The length of the glue coating area is judged to see if it is abnormal; including:

[0038] If the number of glue strip areas is 0, it is determined that the glue coating area detection is abnormal;

[0039] If the number of strip regions is greater than or equal to 2, and Length(L s ,L b )≥D, then the detection of the glue coating area is abnormal; where Ls ,L b The two detection points closest to the adjacent regions, Length(L s ,L b ) is the distance between two points, and D is the preset glue-cutting length range.

[0040] If L i ∈W, W is the preset glue width range, and the glue coating detection in the area is normal.

[0041] like And Length(L i )≥D, the glue coating area is detected abnormally; where Length(L i ) is the length that continuously exceeds the preset glue width range, and D is the preset glue width defect length range.

[0042] A detection system for a gluing process, characterized by comprising:

[0043] Image acquisition module, used to obtain image frames of workpieces in the gluing process in real time;

[0044] An image extraction module is used to locate and extract the rubber strip area in the workpiece image frame by using a convolutional neural network based on deep learning to obtain the rubber strip area;

[0045] The track detection module is used to determine the starting point of the rubber strip, perform track detection on the rubber strip area through the starting point, and obtain the rubber strip track detection point set

[0046] The rubber strip length calculation module is used to convert the discrete trajectory detection point set Ordered so that each pixel C i The nearest neighbor point f i and b i are the two points in front and behind along the length direction of the rubber strip in the actual position; according to each pixel point C i And the two nearest points are obtained by SVD method i The normal direction of the position n i ; Along the normal direction n i The positive and negative directions are searched simultaneously until the edge of the rubber strip is found. The rubber strip is at pixel point C. i The length is L i =a i +b i , where a i , b i is the length of the rubber strip along the positive and negative semi-axis in the normal direction;

[0047] The abnormality judgment module is used to judge the abnormality of the strip area and pixel C. iThe length of the glue coating area can be used to determine whether it is abnormal.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] The present invention obtains the workpiece image frame in the gluing process in real time, and uses a convolutional neural network based on deep learning to quickly locate and extract the glue strip area in the workpiece image frame to obtain the glue strip area, thereby improving the detection efficiency;

[0050] In terms of rubber strip width detection, the present invention orders the discrete trajectory detection point set, ensuring that the relationship between each pixel point and its nearest neighbor point accurately reflects the actual position, providing a reliable basis for subsequent calculations. Then, the SVD method is used to obtain the normal direction of the position based on the pixel point and its two nearest points. Since the normal direction has a stronger correlation with the edge of the rubber strip, it is more adaptable to the actual shape and direction of the rubber strip than the vertical interval detection line method. Therefore, searches are performed simultaneously along the positive and negative directions of the normal direction until the edge of the rubber strip is found. This allows a more comprehensive consideration of the length of the rubber strip at the pixel point. Finally, the length of the rubber strip at the pixel point is determined by calculating the length of the rubber strip along the positive and negative semi-axes in the normal direction, which can more accurately reflect the actual width of the rubber strip and ensure the accuracy of the detection;

[0051] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A schematic diagram of the method of the present invention is shown;

[0054] Figure 2 A schematic diagram showing the use of deep learning to locate the rubber strip area;

[0055] Figure 3 A schematic diagram of real-time acquisition of the rubber strip trajectory point set is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0057] Figure 1 A schematic diagram of the method according to the present invention is shown, and the specific implementation details of the present invention include:

[0058] 1. Step 1: Real-time acquisition of workpiece image frames during the gluing process:

[0059] 2. Step 2: Use a deep learning-based convolutional neural network to locate and extract the rubber strip area in the workpiece image frame. According to the number of rubber strip areas obtained, select the subsequent detection process:

[0060] Specifically include:

[0061] The image is sent to the convolution layer, and the convolution layer uses the convolution part of the RexNet50 network to obtain the rubber strip feature map (the introduced residual module allows the network to not degenerate when the depth increases, and can extract the rubber strip features well, while reducing the training parameters, which is conducive to model deployment).

[0062] The extracted rubber strip feature map is sent to the RPN structure to classify the rubber strip foreground and background into two categories to obtain the anchor frame information of the rubber strip foreground area.

[0063] The regression module is used to correct the position of the anchor frame in the rubber strip foreground area and output the position of the rubber strip area.

[0064] RPN (Region Proposal Network) is a widely used structure in object detection. Its main function is to efficiently generate candidate regions that may contain target objects. In the object detection task, it is necessary to first determine the areas in the image where the target may exist, and then classify these areas and accurately regress the bounding box to determine the category and specific location of the target.

[0065] 3. Step 3: Determine the starting point of the rubber strip, automatically track the rubber strip area through the starting point, and obtain the rubber strip trajectory detection point set in real time:

[0066] Specifically include:

[0067] (1) Determine the starting point of the strip:

[0068] If the number of adhesive strip areas is 0, there is adhesive breakage in the current image frame, so skip this step;

[0069] If the number of glue strip areas is 1, the visual sensor and the glue gun installation positions are relatively fixed, the pixel position of the glue nozzle position in each frame image captured by the camera in real time also remains unchanged, and the glue nozzle position is manually set in advance to the glue strip starting position point L0.

[0070] If the glue strip area is greater than 2, the glue may be broken in the current image frame, and the distance transformation algorithm is used to obtain the maximum distance of each area as the starting point L0 of the glue strip.

[0071] L0=Max(D(p))=Max(Min(disf(p,q)))p∈O,q∈B

[0072] Among them, O is the set of foreground glue road areas, B is the set of background areas, and the distance map is D.

[0073] (2) Automatically track the rubber strip area through the starting point and obtain the rubber strip trajectory detection point set in real time:

[0074] A1. Generate a circular area R0′ at image L0, the center of R0′ coincides with L0, and the radius is the ideal glue width;

[0075] A2. Perform threshold segmentation on the R0′ region to separate the rubber strip foreground from the background image, obtain the centroid C0 of the rubber strip foreground region, generate a circular region R0 of the same size as R0′ at C0, and determine the direction of the rubber strip trajectory generation based on the characteristic that the length of the rubber strip is larger than its width;

[0076] A3: Generate a second circular area along the direction of the rubber strip track at a distance equal to the preset interval point distance. Repeat A2 to obtain a series of rubber strip track detection point sets.

[0077] 4. Step 4: Calculate the width of the rubber strip at the corresponding position based on the obtained rubber strip trajectory point set:

[0078] Specifically include:

[0079] B1. Discrete trajectory detection point set Ordered so that each pixel C i The nearest neighbor point f i and b i are the two front and rear points along the length direction of the rubber strip in the actual position;

[0080] B2. According to each point and its two nearest points, C can be obtained by SVD (singular value decomposition) method. i The normal direction of the position n i ;

[0081] B3, along the normal direction n iThe positive and negative directions are searched simultaneously until the edge of the rubber strip is found. The rubber strip is at pixel point C. i The length is L i =a i +b i , where a i , b i is the length of the rubber strip along the positive and negative semi-axis in the normal direction.

[0082] 5. Step 5: The comprehensive judgment module obtains the glue strip detection result of the glue-coated image and determines whether the glue-coated area is normal:

[0083] Specifically include:

[0084] If Count(region)=0, the glue coating region is detected abnormally (glue break), and Count(region) is the number of glue strip regions.

[0085] If Count(region)≥2 and Length(L a ,L b )≥D,L a ,L b The two detection points closest to the adjacent regions, Length(L a ,L b ) is the distance between the two points, D is the preset glue-breaking length range, and the glue-coating area is detected to be abnormal (glue-breaking).

[0086] If L i ∈W, W is the preset glue width range, and the glue coating detection in the area is normal.

[0087] like And Length(L i )≥D,Length(L i ) is the length that continuously exceeds the preset glue width range, and D is the preset glue width defect length range, then the glue coating area is detected abnormally (glue width defect).

[0088] Based on the method of the present invention, the embodiment of the present disclosure also provides a detection system corresponding to the above method, which includes:

[0089] Image acquisition module, used to obtain image frames of workpieces in the gluing process in real time;

[0090] An image extraction module is used to locate and extract the rubber strip area in the workpiece image frame by using a convolutional neural network based on deep learning to obtain the rubber strip area;

[0091] The track detection module is used to determine the starting point of the rubber strip, perform track detection on the rubber strip area through the starting point, and obtain the rubber strip track detection point set

[0092] The rubber strip length calculation module is used to convert the discrete trajectory detection point set Ordered so that each pixel C i The nearest neighbor point f i and b i are the two points in front and behind along the length direction of the rubber strip in the actual position; according to each pixel point C i And the two nearest points are obtained by SVD method i The normal direction of the position n i ; Along the normal direction n i The positive and negative directions are searched simultaneously until the edge of the rubber strip is found. The rubber strip is at pixel point C. i The length is L i =a i +b i , where a i , b i is the length of the rubber strip along the positive and negative semi-axis in the normal direction;

[0093] The abnormality judgment module is used to judge the abnormality of the strip area and pixel C. i The length of the glue coating area can be used to determine whether it is abnormal.

[0094] Based on the same inventive concept as the above disclosed content, an embodiment of the present disclosure also provides a device corresponding to the above method, which includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method.

[0095] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. An indirect connection method can be applied to the embodiments of the present disclosure as long as the purpose of the present disclosure is achieved.

[0096] Based on the same inventive concept, the present disclosure also provides a computer storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor performs the above method.

[0097] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for detecting a gluing process, characterized in that: include: Real-time acquisition of workpiece image frames during the gluing process; The convolutional neural network based on deep learning is used to locate and extract the rubber strip area in the workpiece image frame to obtain the rubber strip area; Determine the starting point of the rubber strip, perform trajectory detection on the rubber strip area through the starting point, and obtain the rubber strip trajectory detection point set The discrete trajectory detection point set Ordered so that each pixel C i The nearest neighbor point f i and b i are the two points in front and behind along the length direction of the rubber strip in the actual position; according to each pixel point C i And the two nearest points are obtained by SVD method i The normal direction of the position n i ; Along the normal direction n i The positive and negative directions are searched simultaneously until the edge of the rubber strip is found. The rubber strip is at pixel point C. i The length is L i =a i +b i , where a i , b i is the length of the rubber strip along the positive and negative semi-axis in the normal direction; According to the strip area and pixel C i The length of the glue coating area can be used to determine whether it is abnormal.

2. A method for detecting a glue coating process according to claim 1, characterized in that: The convolutional neural network based on deep learning is used to locate and extract the rubber strip area in the workpiece image frame to obtain the rubber strip area; include: Send the image to the convolutional layer to obtain the feature map of the rubber strip, and the convolutional layer is the convolution module of the RexNet50 network; The extracted rubber strip feature map is sent to the RPN structure to classify the rubber strip foreground and background into two categories, and the anchor frame information of the rubber strip foreground area is obtained; The regression module is used to correct the position of the anchor frame in the rubber strip foreground area and output the position of the rubber strip area.

3. The method for detecting a glue coating process according to claim 1, characterized in that: Determining the starting point of the rubber strip comprises: If the number of glue strip areas is 0, it means that there is glue breakage in the current image frame, and the determination of the glue strip starting point is stopped at this time.

4. The method for detecting a glue coating process according to claim 1, characterized in that: Determining the starting point of the rubber strip comprises: If the number of glue strip areas is 1, the glue nozzle position of the glue coating device is set to the glue strip starting position point L0.

5. The method for detecting a glue coating process according to claim 1, characterized in that: Determining the starting point of the rubber strip comprises: If the number of rubber strip regions is not less than 2, the distance transformation algorithm is used to find the maximum distance of each rubber strip region as the starting position point L0 of the rubber strip; L0=Max(D(p))=Max(Min(disf(p,q)))p∈O,q∈B Among them, O is the set of foreground glue road areas, B is the set of background areas, and the distance map is D.

6. A method for detecting a glue coating process according to claim 5, characterized in that: The rubber strip area is tracked by the starting point to obtain a rubber strip track detection point set. include: Generate a circular area R0′ at L0, the center of R0′ coincides with L0, and the radius is the ideal glue width; Perform threshold segmentation on the R0′ region to separate the rubber strip foreground from the background image, find the centroid C0 of the rubber strip foreground region, generate a circular region R0 of the same size as R0′ at C0, and determine the direction of the rubber strip trajectory generation based on the characteristic that the length of the rubber strip is larger than its width; Generate a second circular area along the direction of the rubber strip track at a distance equal to the preset interval point distance. Repeat the operation to obtain a series of rubber strip track detection point sets.

7. The method for detecting a glue coating process according to claim 1, characterized in that: According to the strip area and pixel point C i The length of the glue coating area is used to determine whether it is abnormal; include: If the number of glue strip areas is 0, it is determined that the glue coating area detection is abnormal; If the number of strip regions is greater than or equal to 2, and Length(L a ,L b )≥D, then the detection of the glue coating area is abnormal; where L a ,L b The two detection points closest to the adjacent regions, Length(L a ,L b ) is the distance between two points, and D is the preset glue-cutting length range. If L i ∈W, W is the preset glue width range, and the glue coating detection in the area is normal. like And Length(L i )≥D, the glue coating area is detected abnormally; where Length(L i ) is the length that continuously exceeds the preset glue width range, and D is the preset glue width defect length range.

8. A detection system for glue coating process, characterized in that: include: Image acquisition module, used to obtain image frames of workpieces in the gluing process in real time; An image extraction module is used to locate and extract the rubber strip area in the workpiece image frame by using a convolutional neural network based on deep learning to obtain the rubber strip area; The track detection module is used to determine the starting point of the rubber strip, perform track detection on the rubber strip area through the starting point, and obtain the rubber strip track detection point set The rubber strip length calculation module is used to convert the discrete trajectory detection point set Ordered so that each pixel C i The nearest neighbor point f i and b i are the two points in front and behind along the length direction of the rubber strip in the actual position; according to each pixel point C i And the two nearest points are obtained by SVD method i The normal direction of the position n i ; Along the normal direction n i The positive and negative directions are searched simultaneously until the edge of the rubber strip is found. The rubber strip is at pixel point C. i The length is L i =a i +b i , where a i , b i is the length of the rubber strip along the positive and negative semi-axis in the normal direction; The abnormality judgment module is used to judge the abnormality of the strip area and pixel C. i The length of the glue coating area can be used to determine whether it is abnormal.

9. A detection device for a glue coating process, comprising: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for detecting the gluing process described in any one of claims 1 to 7.

10. A computer storage medium having executable instructions stored thereon, wherein when the instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Gluing detection method, system and device and storage medium

    CN117392060A

  • Diesel engine gluing detection method and device based on 2D machine vision

    CN117392129A

Cited By

  • Gluing length automatic statistical method and system

    CN120235931A