A human-robot collaborative product surface defect detection method and system

Abstract

This invention belongs to the field of product quality inspection technology. It discloses a human-machine collaborative method and system for detecting product surface defects, reducing misjudgments in product defect detection and solving the problem of existing technologies requiring on-site factory work for product verification, which is limited by space and time. In this invention, after acquiring an image of the surface to be inspected of the product, a defect detection algorithm is used to initially screen candidate defects in the image. For products that fail the initial screening, the image and candidate defects are uploaded to a manual verification platform, where humans verify the candidate defects to determine whether they are genuine defects. If one of the candidate defects of a product is a genuine defect, the product is determined to be unqualified; otherwise, if no genuine defects are found among all the candidate defects of the product, the product is determined to be qualified. The verification results are stored in a database, and the sorting equipment queries the manual verification results of the corresponding product using the product's unique ID number and performs sorting.

Description

Technical Field

[0001] This invention belongs to the field of product quality inspection technology, specifically relating to a human-machine collaborative method and system for detecting product surface defects. Background Technology

[0002] In the industrial manufacturing sector, product quality inspection is the most important and critical step at the end of the production process. To ensure the quality of products leaving the factory, companies must detect and process defective products. At the same time, finding defective products and analyzing their various types of defects, their locations, proportions, and sizes are crucial for improving production processes and predictive maintenance.

[0003] Currently, the industry, especially in high-end precision manufacturing, mostly relies on manual visual inspection for product defect detection and post-processing. For some product categories, due to their small size and complex appearance, quality inspectors even need to examine each item under a microscope. For quality inspectors, this involves high workload, monotonous and repetitive tasks, and may also negatively impact their eyesight. Furthermore, training quality inspectors is often time-consuming and labor-intensive, resulting in high labor costs, low efficiency, poor inspection results, and inconsistent product quality.

[0004] With the development of machine vision technology, especially the rise of deep learning, some automated product surface defect detection equipment has emerged in the industry. For simple product defect detection, it can basically replace manual labor, achieving good results in reducing costs and improving product quality for enterprises. However, for complex products, especially those in the high-precision manufacturing industry, machines can still make certain misjudgments, regardless of whether they are using machine vision or deep learning technology. If qualified products are mistakenly identified as unqualified and discarded, it can cause significant losses to enterprises. On the other hand, even if inspection workers can currently verify products, they generally need to work on-site in the factory, which is limited by work space and time. Summary of the Invention

[0005] The technical problem to be solved by this invention is to propose a human-machine collaborative product surface defect detection method and system to reduce misjudgment of product defect detection and solve the problem that product verification in the prior art requires on-site office work in the factory, which is limited by space and time.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] On the one hand, the present invention provides a human-machine collaborative method for detecting surface defects in products, comprising the following steps:

[0008] S1. Obtain images of all surfaces to be inspected of the product under test and bind them to the product's unique ID number;

[0009] S2. Use a defect detection algorithm to inspect the images of all surfaces of the product to be inspected;

[0010] S3. Determine whether there are candidate defects on each surface to be inspected. If so, obtain the candidate defects on the corresponding surface image and proceed to step S4. If there are no candidate defects on all surfaces to be inspected, directly determine that the product is a qualified product and sort it into a qualified pipeline. Then proceed to the inspection process of the next product to be tested and return to step S1.

[0011] S4. Upload the image of the surface to be inspected of the product with candidate defects and the corresponding candidate defect information to the manual review cloud platform via the network.

[0012] S5. The manual review cloud platform marks the candidate defect areas on the image based on the received image of the surface to be inspected and the corresponding candidate defect information.

[0013] S6. Reviewers use smart terminals to review the candidate defect areas marked on the images on the manual review cloud platform, and mark whether the product is qualified or not based on the review results.

[0014] S7. For products that are manually verified, the sorting equipment queries the database for the corresponding tag based on the product's unique ID number and sorts the products according to the tag: products marked as qualified are sorted into the qualified pipeline, and products marked as unqualified are sorted into the unqualified pipeline.

[0015] Furthermore, in step S1, acquiring images of all surfaces to be inspected of the product under test specifically includes:

[0016] Industrial image acquisition equipment is used to photograph all surfaces of the product to be inspected.

[0017] Furthermore, in step S2, the defect detection algorithm includes, but is not limited to: target detection algorithm, anomaly detection algorithm, semantic segmentation algorithm, morphological method and / or template matching method.

[0018] Furthermore, in step S2, the defect detection algorithm is used to detect images of all surfaces of the product to be inspected, specifically including:

[0019] A semantic segmentation detection model is used to detect the image of the surface to be detected, generating a binary image to distinguish between defective and non-defective regions. Morphological operations are then performed on the binary image to obtain candidate defects in the surface image to be detected.

[0020] Furthermore, in step S6, the step of reviewing the candidate defect areas marked on the image and marking whether the product is qualified or not based on the review results specifically includes:

[0021] Determine whether each candidate defect is a real defect; if at least one real defect exists among all the candidate defects for the surfaces to be inspected uploaded by the product, mark the product as unqualified; if none of the candidate defects for the surfaces to be inspected are real defects, mark the product as qualified; record the marking results in the database.

[0022] On the other hand, the present invention also provides a human-machine collaborative product surface defect detection system, comprising:

[0023] The image acquisition module is used to acquire images of all surfaces to be inspected of the product under test and bind them to the product's unique ID number.

[0024] The defect detection module is used to detect candidate defects based on images of all surfaces of the product to be inspected obtained by the image acquisition module and a defect detection algorithm.

[0025] The execution module is used to perform initial screening of products based on the detection results of the defect detection module. If candidate defects are found, the product is initially screened as unqualified, and images of the candidate defects and the detection surfaces where the defects are located are uploaded. If no candidate defects are found on any of the detection surfaces, the product is judged as qualified.

[0026] The manual review module is used to mark candidate defect areas on images of products initially screened as unqualified, so that manual reviewers can review the marked candidate defect areas on the images and mark whether the products are qualified or not based on the review results. The qualification and non-qualification marks of the manually reviewed products are stored in the database.

[0027] The sorting module uses the unique ID number of the manually verified product to query the database for the verification result tag, and sorts the products according to the tag: products marked as qualified are sorted into qualified channels, and products marked as unqualified are sorted into unqualified channels.

[0028] Furthermore, the manual review module is also used to maintain user information, manage roles and permissions, query and statistically analyze task completion status, set task status, and view details of completed tasks.

[0029] Furthermore, the roles managed by the manual review module are divided into ordinary users, team administrators, quality administrators, and super administrators; permission management includes: the permissions of ordinary users are configured by team administrators and super administrators; the permissions of team administrators are configured by super administrators; the permissions of quality administrators are configured by super administrators; the super administrator is built into the system and has all permissions.

[0030] The beneficial effects of this invention are:

[0031] The solution provided by this invention combines human and machine collaboration. Products initially deemed unqualified by the machine are then manually reviewed by a cloud-based human verification platform. If the manual review reveals at least one genuine defect, the product is marked as unqualified; otherwise, it is marked as qualified. All verification can be completed on smart terminals such as computers, mobile phones, and tablets. Since defects in unqualified products are often few, only a small number of images require manual review. This approach combines the efficiency of machine-based surface defect detection with the potential for misjudgment by machines. Furthermore, the cloud-based human verification platform allows workers to review unqualified images from anywhere with internet access, such as at home or in the office, without needing to visit the factory for on-site inspection, thus solving the problems of limited space and time. Attached Figure Description

[0032] Figure 1 This is a flowchart of the human-machine collaborative product surface defect detection method in an embodiment of the present invention;

[0033] Figure 2 This is a structural block diagram of a human-machine collaborative product surface defect detection system in an embodiment of the present invention. Detailed Implementation

[0034] This invention aims to propose a human-machine collaborative method and device for detecting product surface defects, reducing misjudgments in product defect detection and solving the problem of existing technologies requiring on-site factory work for product verification, which is limited by space and time. The core idea is as follows: After acquiring an image of the surface of the product to be inspected, a defect detection algorithm is used to initially screen candidate defects in the image. For products that fail the initial screening, the image and candidate defects are uploaded to a manual verification platform, where humans verify the candidate defects to determine if they are genuine defects. If any candidate defect of a product is a genuine defect, the product is determined to be unqualified; otherwise, if no genuine defects are found among all candidate defects, the product is determined to be qualified. The verification results are stored in a database, and the sorting equipment queries the corresponding manual verification result for the product using its unique ID number and performs sorting.

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Example 1:

[0037] This embodiment is an example of a human-machine collaborative product surface defect detection method, such as... Figure 1 As shown, the method includes:

[0038] S1. Obtain images of all surfaces to be inspected of the product under test and bind them to the product's unique ID number;

[0039] In this step, images of all surfaces of a product to be inspected are acquired using industrial image acquisition equipment. Multiple images can be acquired for each surface based on product and defect characteristics. All images are bound to a unique ID number for the product. This unique ID number can be a number, string, QR code, etc., used to distinguish different products.

[0040] For example, if the product under test is a connector, a method for obtaining an image of the appearance inspection area of ​​the surface to be inspected of a connector product includes: using an image acquisition device to photograph all surfaces to be inspected of a product, including: the base surface, the pin surface, the end protector surface, the fisheye surface, and the pin tip and fisheye tip.

[0041] S2. Use a defect detection algorithm to inspect the images of all surfaces of the product to be inspected;

[0042] In this step, the defect detection algorithm includes, but is not limited to: target detection algorithm, anomaly detection algorithm, semantic segmentation algorithm, morphological method and / or template matching method. One specific implementation involves using a semantic segmentation detection model to detect the image of the surface to be detected, generating a binary image, distinguishing between defective and non-defective regions, and performing morphological operations on the binary image to obtain candidate defects in the surface image.

[0043] S3. Determine whether there are candidate defects on each surface to be inspected. If so, obtain the candidate defects on the corresponding surface image and proceed to step S4. If there are no candidate defects on all surfaces to be inspected, directly determine that the product is a qualified product and sort it into a qualified pipeline. Then proceed to the inspection process of the next product to be tested and return to step S1.

[0044] In this step, if there is at least one candidate defect on all the surfaces to be inspected of the product, the product is initially screened as an unqualified product; otherwise, if there are no candidate defects on any of the surfaces to be inspected of the product, the product is determined to be a qualified product.

[0045] S4. Upload the image of the surface to be inspected of the product with candidate defects and the corresponding candidate defect information to the manual review cloud platform via the network.

[0046] In this step, in order to further determine the candidate defects of the initially screened unqualified products, the images of these candidate defects and their corresponding inspection surfaces can be uploaded to the manual review cloud platform via the 5G network. Of course, the unique ID number of the product needs to be uploaded for product identification.

[0047] The manual review cloud platform includes a front-end, a back-end, and a database. Users can accept tasks online and view details of completed tasks through the front-end. The back-end is an operation and management platform where administrators maintain user information, manage roles and permissions, query and statistically analyze task completion status, and set task status. The database is used to store image information of surfaces to be inspected, candidate defect information, and manual review information.

[0048] The user types for the manual review platform are differentiated into ordinary users, team administrators, quality administrators, and super administrators. Ordinary users have permissions configured by team administrators and super administrators; team administrators have permissions configured by super administrators; quality administrators have permissions configured by super administrators; and super administrators are built into the system and have all permissions.

[0049] S5. The manual review cloud platform marks the candidate defect areas on the image based on the received image of the surface to be inspected and the corresponding candidate defect information.

[0050] In this step, for the images of the inspection surfaces of products that are initially screened as unqualified, the candidate defect areas are drawn on the images according to the algorithm results, thereby marking the candidate defect areas and facilitating subsequent manual review of the candidate defect areas.

[0051] S6. Reviewers use smart terminals to review the candidate defect areas marked on the images on the manual review cloud platform, and mark whether the product is qualified or not based on the review results.

[0052] In this step, reviewers can use smart terminals such as computers, mobile phones, and tablets to review the candidate defect areas marked on the images on a manual review cloud platform. Specifically, based on product quality inspection standards, personnel judge whether each frame contains a real defect by examining the images on the terminal, thereby verifying whether the product is qualified. For example, if the personnel determine that there are no abnormalities within the frames on the base surface image, the candidate defect is considered a false defect, and if there are no candidate defect frames in other surface images, the product is considered qualified.

[0053] Finally, the reviewers record the manual review results (marks indicating whether the product is qualified or unqualified) in the database, along with the unique ID number of the corresponding product.

[0054] S7. For products that are manually verified, the sorting equipment queries the database for the corresponding tag based on the product's unique ID number and sorts the products according to the tag: products marked as qualified are sorted into the qualified pipeline, and products marked as unqualified are sorted into the unqualified pipeline.

[0055] In this step, the sorting equipment can obtain the unique ID number of the product by identifying the product label, then query the manual review results in the database of the manual review platform based on the unique ID number of the product, and finally sort the products according to the manual review results. The sorting results can be qualified, unqualified but repairable, unqualified and unrepairable, unprocessed, etc.

[0056] Example 2:

[0057] This embodiment is an example of a human-machine collaborative product surface defect detection system, such as... Figure 2 As shown, the system includes:

[0058] The image acquisition module is used to acquire images of all surfaces to be inspected of the product under test and bind them to the product's unique ID number.

[0059] The defect detection module is used to detect candidate defects based on images of all surfaces of the product to be inspected obtained by the image acquisition module and a defect detection algorithm.

[0060] The execution module is used to perform initial screening of products based on the detection results of the defect detection module. If candidate defects are found, the product is initially screened as unqualified, and images of the candidate defects and the detection surfaces where the defects are located are uploaded. If no candidate defects are found on any of the detection surfaces, the product is judged as qualified.

[0061] The manual review module is used to mark candidate defect areas on images of products initially screened as unqualified, allowing for manual review of these marked areas. Based on the review results, the module marks whether the products are qualified or not, and stores these marks in a database. The manual review module also maintains user information, manages roles and permissions, queries and statistically analyzes task completion, sets task status, and displays details of completed tasks.

[0062] The sorting module uses the unique ID number of the manually verified product to query the database for the verification result tag, and sorts the products according to the tag: products marked as qualified are sorted into qualified channels, and products marked as unqualified are sorted into unqualified channels.

[0063] Since the sum of the functions implemented by each module of this system corresponds to the human-machine collaborative product surface defect detection method in Example 1, the specific implementation methods of each module will not be described in detail in this example.

[0064] Finally, it should be noted that the above embodiments are merely preferred embodiments and are not intended to limit the present invention. It should be pointed out that those skilled in the art can make various modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the claims, and all such modifications, substitutions, and improvements should be included within the scope of protection of the present invention.

Claims

1. A human-machine collaborative method for detecting surface defects in products, characterized in that, Includes the following steps: S1. Obtain images of all surfaces to be inspected of the product under test and bind them to the product's unique ID number; S2. Use a defect detection algorithm to inspect the images of all surfaces of the product to be inspected; S3. Determine whether there are candidate defects on each surface to be inspected. If so, obtain the candidate defects on the corresponding surface image and proceed to step S4. If there are no candidate defects on all surfaces to be inspected, directly determine that the product is a qualified product and sort it into a qualified pipeline. Then proceed to the inspection process of the next product to be tested and return to step S1. S4. Upload the image of the surface to be inspected of the product with candidate defects and the corresponding candidate defect information to the manual review cloud platform via the network. S5. The manual review cloud platform marks the candidate defect areas on the image based on the received image of the surface to be inspected and the corresponding candidate defect information. S6. Reviewers use smart terminals to review the candidate defect areas marked on the images on the manual review cloud platform, and mark whether the product is qualified or not based on the review results. The process of reviewing the candidate defect regions marked on the image and marking the product as qualified or unqualified based on the review results specifically includes: Determine whether each candidate defect is a real defect; If at least one actual defect exists among all the candidate defects for the surfaces to be inspected uploaded by the product, the product is marked as unqualified; if none of the candidate defects for the surfaces to be inspected are actual defects, the product is marked as qualified; the marking results are recorded in the database. S7. For products that are manually verified, the sorting equipment queries the database for the corresponding tag based on the product's unique ID number and sorts the products according to the tag: products marked as qualified are sorted into the qualified pipeline, and products marked as unqualified are sorted into the unqualified pipeline.

2. The human-machine collaborative product surface defect detection method as described in claim 1, characterized in that, In step S1, acquiring images of all surfaces to be inspected of the product under test specifically includes: Industrial image acquisition equipment is used to photograph all surfaces of the product to be inspected.

3. The human-machine collaborative product surface defect detection method as described in claim 1, characterized in that, In step S2, the defect detection algorithm includes, but is not limited to: target detection algorithm, anomaly detection algorithm, semantic segmentation algorithm, morphological method and / or template matching method.

4. The human-machine collaborative product surface defect detection method as described in claim 3, characterized in that, The defect detection algorithm is used to detect images of all surfaces of the product to be inspected, specifically including: A semantic segmentation detection model is used to detect the image of the surface to be detected, generating a binary image to distinguish between defective and non-defective regions. Morphological operations are then performed on the binary image to obtain candidate defects in the surface image to be detected.

5. A human-machine collaborative product surface defect detection system, characterized in that, include: The image acquisition module is used to acquire images of all surfaces to be inspected of the product under test and bind them to the product's unique ID number. The defect detection module is used to detect candidate defects based on images of all surfaces of the product to be inspected obtained by the image acquisition module and a defect detection algorithm. The execution module is used to perform initial screening of products based on the detection results of the defect detection module. If candidate defects are found, the product is initially screened as unqualified, and images of the candidate defects and the detection surfaces where the defects are located are uploaded. If no candidate defects are found on any of the detection surfaces, the product is judged as qualified. The manual review module is used to mark candidate defect areas on images of products initially screened as unqualified, so that manual reviewers can review the marked candidate defect areas on the images and mark whether the products are qualified or not based on the review results. The qualification and non-qualification marks of the manually reviewed products are stored in the database. The process of reviewing the candidate defect regions marked on the image and marking the product as qualified or unqualified based on the review results specifically includes: Determine whether each candidate defect is a real defect; if at least one real defect exists among all the candidate defects for the surfaces to be inspected uploaded by the product, mark the product as unqualified; if none of the candidate defects for the surfaces to be inspected are real defects, mark the product as qualified; record the marking results in the database. The sorting module uses the unique ID number of the manually verified product to query the database for the verification result tag, and sorts the products according to the tag: products marked as qualified are sorted into qualified channels, and products marked as unqualified are sorted into unqualified channels.

6. The human-machine collaborative product surface defect detection system as described in claim 5, characterized in that, The manual review module is also used to maintain user information, manage roles and permissions, query and statistically analyze task completion status, set task status, and view details of completed tasks.

7. The human-machine collaborative product surface defect detection system as described in claim 6, characterized in that, The roles managed by the manual review module are divided into ordinary users, team administrators, quality administrators, and super administrators; permission management includes: the permissions of ordinary users are configured by team administrators and super administrators; the permissions of team administrators are configured by super administrators; the permissions of quality administrators are configured by super administrators; the super administrator is built into the system and has all permissions.

Images