Product defect visual inspection system and method

Abstract

The invention relates to a product defect visual inspection system, which comprises a first acquisition device for acquiring a front low-exposure image of a product, a second acquisition device for acquiring a front high-exposure image of the product, and a control device for determining a grabbing pose according to the front low-exposure image or / and the front high-exposure image so as to control a grabbing device to grab the product. The third acquisition device is used for acquiring a back low-exposure image and a back high-exposure image of the product after the product is grabbed, and the control device is also used for identifying a front scratch defect according to the front low-exposure image and identifying a front smudginess defect according to the front high-exposure image; and recognizing a back dirt defect according to the back low-exposure image, and recognizing a back scratch defect according to the back high-exposure image. The problems that an existing monitoring system is low in accuracy and consistency of detection results, cannot adapt to complex and changeable defect characteristics of products and is low in detection precision of small-size defects are solved.

Description

Technical Field

[0001] The present invention relates to the field of visual inspection technology, and in particular to a product defect visual inspection system and method. Background Art

[0002] In the production process of products, product quality assurance is crucial, especially the detection of surface defects of products. At present, there are two main methods for product defect detection: manual detection and single-lighting single-station single-stage visual inspection.

[0003] Manual inspection relies on the subjective judgment of quality inspectors. Since each person has different recognition experience and judgment standards, the inspection results are highly subjective and inconsistent. In addition, manual inspection is inefficient and cannot meet the needs of large-scale production. When faced with products of various shapes, various defects and small sizes, manual inspection is prone to false detection and missed detection.

[0004] Although the single-lighting single-station single-stage visual inspection has improved the accuracy and efficiency of inspection to a certain extent, it still has many shortcomings. First, the single-lighting visual inspection system cannot adapt to the complex defect types on the product surface and can only identify a single defect type. Secondly, the single-station inspection system cannot meet the needs of large-scale production in terms of production rhythm. In addition, the single-stage inspection system is difficult to achieve ideal accuracy when identifying smaller defects.

[0005] Therefore, how to improve it is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0006] Based on this, the purpose of this application is to provide a product defect visual inspection system and method to solve at least one of the technical problems mentioned in the above background technology.

[0007] In a first aspect, the present application provides a product defect visual inspection system, comprising: a transport device; a first acquisition device, a second acquisition device, and a third acquisition device arranged along the travel direction of the transport device, as well as a grasping device and a control device; during product transportation:

[0008] The first acquisition device is used to acquire a low-exposure image of the front side of the product;

[0009] The second acquisition device is used to acquire a high-exposure image of the front side of the product;

[0010] A control device, used to determine a grasping posture according to the front low-exposure image and / or the front high-exposure image, so as to control the grasping device to grasp the product;

[0011] A third acquisition device is used to acquire a low-exposure image and a high-exposure image of the back side of the product after the product is grabbed;

[0012] The control device is also used to identify front scratch defects based on the front low-exposure image and identify front dirt defects based on the front high-exposure image; identify back dirt defects based on the back low-exposure image and identify back scratch defects based on the back high-exposure image.

[0013] Furthermore, the control device comprises:

[0014] An image processing unit, used to perform instance segmentation on the image to obtain product contours, and obtain the minimum circumscribed rectangle of each product contour, so as to obtain product size data according to the minimum circumscribed rectangle of the product contour; the product size data includes product width and height, product angle, product center point, and product vertex;

[0015] A posture calculation unit, used to determine the grasping posture according to the product size data;

[0016] The gripping control unit is used to control the gripping device to grip the product according to the gripping posture.

[0017] Furthermore, the image processing unit comprises:

[0018] A segmentation component for performing instance segmentation on the image to obtain product contours;

[0019] A dimension data calculation component is used to obtain the minimum circumscribed rectangle of each product outline, so as to obtain product dimension data according to the minimum circumscribed rectangle of the product outline;

[0020] A filter component is used to filter the minimum bounding rectangle of products whose edges are blocked according to a prefabricated edge mask template.

[0021] Furthermore, the control device further includes:

[0022] A pixel expansion unit is used to expand the pixels of the minimum circumscribed rectangle of the product in the front low-exposure image, the front high-exposure image, the back low-exposure image, and the back high-exposure image, so as to obtain the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image corresponding to the minimum circumscribed rectangle of each product;

[0023] A first defect recognition unit, configured to recognize a front scratch defect based on the front low-exposure cut-out image, or recognize a back dirt defect based on the back low-exposure cut-out image;

[0024] The second defect recognition unit is used to recognize the front dirt defect based on the front high-exposure cut-out image, or to recognize the back scratch defect based on the back high-exposure cut-out image.

[0025] Furthermore, the system also includes:

[0026] The control device is also used to obtain the number of holes on the surface of the product based on the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image and the back high-exposure cut-out image, so as to judge whether the product is qualified based on the scratch defects, dirt defects and the number of holes on the front and back of the product, and classify and store the products according to the judgment results.

[0027] Furthermore, the control device further includes:

[0028] The first defect recognition unit or / and the second defect recognition unit are further used to obtain the number of holes on the surface of the product according to the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image and the back high-exposure cut-out image;

[0029] Area division unit, used to divide the front and back of the product into functional areas and non-functional areas;

[0030] A judging unit, used to judge whether the number of defects contained in the functional area and the non-functional area is greater than a corresponding number threshold and whether the number of holes on the surface of the product is not equal to the hole number threshold, if so, the product is an unqualified product, if not, the product is a qualified product;

[0031] The classification unit is used to grab unqualified products and place them in the first storage area, and grab qualified products and place them in the second storage area.

[0032] In a second aspect, the present application provides a product defect visual detection method, which is applied to a product defect visual detection system as described in the first aspect or any one of the implementations thereof, comprising:

[0033] Collect low-exposure images and high-exposure images of the front of the product;

[0034] Determine a grasping posture according to the front low-exposure image and / or the front high-exposure image to control a grasping device to grasp the product;

[0035] After the product is captured, a low-exposure image of the back side and a high-exposure image of the back side of the product are collected;

[0036] The front scratch defect is identified based on the front low-exposure image, and the front dirt defect is identified based on the front high-exposure image; the back dirt defect is identified based on the back low-exposure image, and the back scratch defect is identified based on the back high-exposure image.

[0037] Further, the step of determining the grasping posture according to the front low-exposure image and / or the front high-exposure image to control the grasping device to grasp the product includes:

[0038] Build and train a neural network model that takes low-exposure images or high-exposure images as input and product outlines as output, which is an instance segmentation model;

[0039] Input the low-exposure image of the front of the product and the high-exposure image of the front of the product into the instance segmentation model to obtain the product contour in the image, and obtain the minimum bounding rectangle of each product contour, so as to obtain the product size data according to the minimum bounding rectangle of the product contour; the product size data includes the product width and height, product angle, product center point, and product vertex;

[0040] The grasping posture is determined according to the product size data, so as to control the grasping device to grasp the product according to the grasping posture.

[0041] Further, the steps of identifying front scratch defects according to the front low-exposure image and identifying front dirt defects according to the front high-exposure image; identifying back dirt defects according to the back low-exposure image and identifying back scratch defects according to the back high-exposure image include:

[0042] Construct and train a neural network model that takes low-exposure cut-out images as input and outputs defect locations and hole numbers as the first defect recognition model;

[0043] Construct and train a neural network model that takes high-exposure cut-out images as input and outputs defect locations and hole numbers as the second defect recognition model;

[0044] Input the low-exposure image of the back of the product and the high-exposure image of the back of the product into the instance segmentation model to obtain the product outlines in the image and obtain the minimum circumscribed rectangle of each product outline;

[0045] Expand the pixels of the minimum bounding rectangles of the products in the front low-exposure image, the front high-exposure image, the back low-exposure image, and the back high-exposure image to obtain the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image corresponding to the minimum bounding rectangles of each product;

[0046] Inputting the front low-exposure cut-out image and the back low-exposure cut-out image into a first defect recognition model to obtain a front scratch defect and a back dirt defect;

[0047] Inputting the front high-exposure cut-out image and the back high-exposure cut-out image into a second defect recognition model to obtain a front dirt defect and a back scratch defect;

[0048] Furthermore, the method further comprises:

[0049] Input the front low-exposure cut-out image and the back low-exposure cut-out image into the first defect recognition model, and input the front high-exposure cut-out image and the back high-exposure cut-out image into the second defect recognition model to obtain the number of holes on the product surface;

[0050] Divide the product surface into functional and non-functional areas;

[0051] Determine whether the number of holes on the product surface is not equal to the hole number threshold. If so, it is an unqualified product. If not, then:

[0052] Determine whether the number of defects contained in the functional area is greater than the first defect number threshold. If so, the product is a non-conforming product. If not, then:

[0053] It is determined whether the number of defects contained in the non-functional area is greater than a second defect number threshold value. If so, the product is an unqualified product; if not, the product is a qualified product.

[0054] The present invention provides a product defect visual inspection system and method, comprising: a transport device; a first acquisition device, a second acquisition device and a third acquisition device arranged along the travel direction of the transport device, as well as a grasping device and a control device; during the product transportation process, the first acquisition device acquires a low-exposure image of the front side of the product, the second acquisition device acquires a high-exposure image of the front side of the product, the control device determines the grasping posture according to the low-exposure image of the front side and / or the high-exposure image of the front side to control the grasping device to grasp the product, the third acquisition device acquires a low-exposure image and a high-exposure image of the back side of the product after the product is grasped, the control device then identifies the front scratch defect according to the low-exposure image of the front side and identifies the front dirt defect according to the high-exposure image of the front side; identifies the back dirt defect according to the low-exposure image of the back side and identifies the back scratch defect according to the high-exposure image of the back side, thereby improving the detection technology, reducing the false detection and missed detection caused by subjective judgment, improving the accuracy and consistency of the detection results, and being able to identify a variety of defect types at the same time to adapt to the complex and changeable defect characteristics of the product, and optimizing the structure and process of the detection system to meet the needs of large-scale production, improve production efficiency, and improve the detection accuracy of small-size defects. It solves the problems of low accuracy and consistency of detection results of existing monitoring systems, inability to adapt to complex and changeable defect characteristics of products, and low detection accuracy for small-size defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a structural schematic diagram of a multi-lighting, multi-station, dual-stage product visual inspection system according to an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the structure of a product defect visual inspection system according to an embodiment of the present invention;

[0057] Figure 3 Schematic diagram of an edge mask template in a filter element according to an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of a stencil template filtering image corresponding to a first defect recognition unit according to an embodiment of the present invention;

[0059] Figure 5A schematic diagram of a stencil template filtering image corresponding to a second defect recognition unit according to an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of a region division template in a region division unit according to an embodiment of the present invention;

[0061] Figure 7 Flow chart of a method for visually detecting product defects according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., then the directional indication is only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly. In addition, if the embodiments of the present invention involve descriptions of "first, second", "S1, S2", "step one, step two", etc., then such descriptions are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features or indicating the execution order of the method, etc. Those skilled in the art can understand that anything that does not violate the gist of the invention under the technical concept of the invention should be included in the protection scope of the present invention.

[0064] Since the single-lighting and single-station inspection method has many defects, such as fixed lighting cannot meet the imaging of different defect types, and thus cannot meet the actual defect detection needs of customers, and single-station cannot meet the product characteristics, cannot meet the characteristics of the conveyor belt, and cannot meet the beat requirements, a multi-lighting multi-station dual-stage product visual inspection system is optional. Figure 1 As shown, it includes station 1, station 2, station 3, and station 4. The pallet product passes through each station in sequence along the direction of the conveyor belt, so that each station collects images of the pallet product in sequence. Among them, station 1 includes a darkroom, two bar light sources, and an industrial camera, which is an example of the first acquisition device. This station is responsible for the front detection of the pallet. The camera will collect an image, which is specifically responsible for the scratch defect on the front of the pallet. The scratch defect includes three sub-defects: scratches, scuffs, and bruises.

[0065] Station 2 includes a darkroom, two bar light sources and an industrial camera, which is an example of the second acquisition device. This station is responsible for the front inspection of the sheet. The camera will capture an image and specifically detect the major category of dirt defects on the front of the sheet. The major category of dirt includes three minor categories of defects: dirt, oil, and black spots.

[0066] Station 3 includes a robot, a manipulator, four suction cups, four strip light sources and an industrial camera, which is responsible for the back detection of the sheet and is an example of the third acquisition device. In this example, since the grasping position takes longer, multiple stations of the same type can be optionally configured, such as station 3 and station 4, to adapt to the assembly line speed of station 1 and station 2. Figure 1 As shown, the manipulators at stations 3 and 4 will absorb 0 to 4 pieces, move to the shooting area of ​​the corresponding station, and then use two lighting methods to collect two images to detect and identify two major types of defects. It is worth noting that the components of stations 3 and 4 can be the same, and the same stations can be added or reduced according to the product inspection speed.

[0067] After the bar product has been inspected at all workstations, four images will be generated, two on the front and two on the back. Finally, based on the recognition results of the four images, the robot will comprehensively determine whether the product is NG or OK, and then place it in the corresponding placement area.

[0068] like Figure 2 As shown, the present invention also provides a product defect visual inspection system, comprising: a transport device 101; a first acquisition device 102, a second acquisition device 103 and a third acquisition device 104 arranged along the travel direction of the transport device 101, as well as a gripping device 105 and a control device 106;

[0069] Specifically, the transport device 101 can be any transport device capable of transporting products, such as a conveyor belt, for uninterrupted transport of products, and other devices are arranged in sequence along the travel direction of the transport device to perform visual inspection and classification of the products.

[0070] Specifically, the first acquisition device 102 is used to acquire a low-exposure image of the front side of the product.

[0071] As an example, the first acquisition device 102 may optionally include a darkroom, two bar light sources and an industrial camera, and use a low exposure method to make the overall imaging a dark field, which is darker, and the scratch defect imaging is white, which is brighter, so as to facilitate the subsequent identification of scratch defects; scratch defects include scratches, scuffs, bruises and other similar defects.

[0072] Specifically, the second acquisition device 103 is used to acquire a high-exposure image of the front side of the product.

[0073] As an example, the second acquisition device 103 may optionally include a darkroom, two strip light sources and an industrial camera, and use a low exposure method so that the overall image is a bright field, which is brighter, and the dirt defect image is black, which is darker, so as to facilitate the subsequent identification of the dirt defects; dirt defects include dirt, oil stains, black spots and other similar defects.

[0074] Specifically, the grasping device 105 is used to grasp the corresponding product according to the grasping posture determined by the control device 106.

[0075] As an example, the grasping device may be a robot arm including four suction cups.

[0076] Specifically, the control device 106 is used to determine the grasping posture according to the front low-exposure image and / or the front high-exposure image, so as to control the grasping device 105 to grasp the product.

[0077] In one possible implementation, the control device 106 includes an image processing unit, which is used to perform instance segmentation on the image to obtain product contours, and obtain the minimum circumscribed rectangle of each product contour to obtain product size data based on the minimum circumscribed rectangle of the product contour; product size data, including product width and height, product angle, product center point, and product vertex; a posture calculation unit, which is used to determine a grasping posture based on the product size data; and a grasping control unit, which is used to control a grasping device to grasp the product based on the grasping posture.

[0078] In one possible implementation, the image processing unit includes a segmentation element for performing instance segmentation on the image to obtain product contours; a size data calculation element for obtaining the minimum enclosing rectangle of each product contour to obtain product size data based on the minimum enclosing rectangle of the product contour; and a filtering element for filtering the minimum enclosing rectangle of the product with obscured edges based on a prefabricated edge mask template.

[0079] As an example, the segmentation component may optionally use the mask-_rcnn_R_50_FPN_3x model as the initial neural network model, collect a number of low-exposure images or / and high-exposure images in advance, and annotate the product contours in each image to construct a training set, and then set the model training parameters to input the annotated low-exposure images or / and high-exposure images into the initial neural network model, and train the initial neural network model according to the set model training parameters to obtain a trained neural network model, wherein the model training parameters may be optionally set as:

[0080] Number of classes (num_classes): 1

[0081] Class name (classes_name): ['bg']

[0082] Batch size (batch_size): 32

[0083] Learning rate: 0.0025

[0084] Epochs: 20000

[0085] Save interval (save_period): 5000

[0086] Maximum size (max_size): 512

[0087] Minimum size (min_size): [512,]

[0088] Among them, bg represents the product, the maximum size is to scale the longest side of the input image to 512, and the minimum size is to scale the shortest side proportionally according to the scaling ratio of the longest side.

[0089] As an example, the dimension data calculation component can optionally use the cv::minAreaRect() function in the opencv library. By inputting the product outline into the cv::minAreaRect() function, the minimum enclosing rectangle of the corresponding product outline can be obtained, and the product dimension data such as product width and height, product angle, product center point, product vertex, etc. can be obtained according to the vertices and center point of the minimum enclosing rectangle. More specifically, the product angle can be selected as the angle between the longest side or midline of the minimum enclosing rectangle of the product outline and an arbitrarily set horizontal line. The specific calculation method can be arbitrarily set by those skilled in the art.

[0090] As an example, the filter element may optionally include Figure 3 The edge mask template shown, in which the black area is the detection area and the white area is the edge area, aligns the edge mask template with the low-exposure image, and determines whether the products contained in each low-exposure image have intersections with the white area. If so, filtering is required, otherwise, it remains unchanged.

[0091] Specifically, the control device 106 also includes a pixel expansion unit, which is used to expand the pixels of the minimum circumscribed rectangle of the product in the front low-exposure image, the front high-exposure image, the back low-exposure image and the back high-exposure image, so as to obtain the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image and the back high-exposure cut-out image corresponding to the minimum circumscribed rectangle of each product; a first defect recognition unit, which is used to recognize the front scratch defect according to the front low-exposure cut-out image, or recognize the back dirt defect according to the back low-exposure cut-out image; a second defect recognition unit, which is used to recognize the front dirt defect according to the front high-exposure cut-out image, or recognize the back scratch defect according to the back high-exposure cut-out image.

[0092] As an example, when the pixel expansion unit performs pixel expansion on the minimum bounding rectangle of each product, it can optionally obtain an area with a size of 256 pixels extending in all directions with each minimum bounding rectangle as the center, and extract the area from the low-exposure image or the high-exposure image, so as to obtain the low-exposure cut-out image or the high-exposure cut-out image corresponding to the minimum bounding rectangle of each product.

[0093] As an example, the first defect recognition unit may optionally use the mask_rcnn_R_50_FPN_3x model as the initial neural network model, obtain a number of low-exposure cut-image images in advance, and mark the holes and defect positions in each low-exposure cut-image image to construct a training set, and then set the model training parameters to input the marked low-exposure cut-image image into the initial neural network model, and train the initial neural network model according to the set model training parameters to obtain a trained neural network model, wherein the model training parameters may be optionally set as:

[0094] Number of classes (num_classes): 4

[0095] Class names (classes_name): ['bg', 'hole', 'quexain1']

[0096] Batch size (batch_size): 32

[0097] Learning rate: 0.0025

[0098] Epochs: 20000

[0099] Save interval (save_period): 5000

[0100] Maximum size (max_size): 640

[0101] Minimum size (min_size): [640,]

[0102] Among them, bg represents the product, hole represents the hole, quexain1 represents the defect position in the low-exposure cut image, the maximum size is to scale the longest side of the input image to 640, and the minimum size is to scale the shortest side proportionally according to the scaling ratio of the longest side.

[0103] More specifically, due to the interference of ambient light when acquiring the low-exposure image of the back side, when the input of the first defect recognition unit is the low-exposure cut-out image of the front side, the output is a front scratch defect; when the input of the first defect recognition unit is the low-exposure cut-out image of the back side, the output is a back dirt defect.

[0104] Preferably, the first defect identification unit may also be arbitrarily set according to those skilled in the art to identify other defect categories such as special categories of faults.

[0105] As an example, the second defect recognition unit may optionally use the mask_rcnn_R_50_FPN_3x model as the initial neural network model, obtain several high-exposure cut-image images in advance, and mark the holes and defect positions in each high-exposure cut-image to construct a training set, and then set the model training parameters to input the marked high-exposure cut-image images into the initial neural network model, and train the initial neural network model according to the set model training parameters to obtain a trained neural network model, wherein the model training parameters may be optionally set as:

[0106] Number of classes (num_classes): 4

[0107] Class names (classes_name): ['bg', 'hole', 'zangwu', 'quexain2']

[0108] Batch size (batch_size): 32

[0109] Learning rate: 0.0025

[0110] Epochs: 20000

[0111] Save interval (save_period): 5000

[0112] Maximum size (max_size): 640

[0113] Minimum size (min_size): [640,]

[0114] Among them, bg represents the product, hole represents the hole, quexain2 represents the defect position in the high-exposure cut image, the maximum size is to scale the longest side of the input image to 640, and the minimum size is to scale the shortest side proportionally according to the scaling ratio of the longest side.

[0115] More specifically, due to the interference of ambient light when obtaining the high-exposure image of the back side, when the input of the second defect recognition unit is the high-exposure cut-out image of the front side, the output is a front dirt defect; when the input of the second defect recognition unit is the high-exposure cut-out image of the back side, the output is a back scratch defect.

[0116] Preferably, the second defect identification unit may also be arbitrarily set according to those skilled in the art to identify other defect categories such as special categories of faults.

[0117] Preferably, since customers have inconsistent defect definition standards, the imprint template filtering image, area threshold, shortest edge threshold and gradient threshold can be pre-set according to the customer's defect definition standard to perform imprint template filtering, pixel area filtering, pixel short edge filtering and gradient filtering on the recognition results of the first defect recognition unit and the second defect recognition unit to remove misidentified defects.

[0118] More specifically, the template filter image can be selected as follows Figure 4 and Figure 5 As shown, Figure 4 Corresponding to the recognition result of the first defect recognition unit, Figure 5 Corresponding to the recognition result of the second defect recognition unit, the black area is the detection area, the white area is the stencil area, and if the minimum circumscribed rectangle of the defect intersects with the white area, filtering is required.

[0119] The pixel area filtering threshold is set to 400, that is, the length of the minimum circumscribed rectangular border of the detected defect multiplied by the width. If it is less than 400, it is filtered.

[0120] The pixel short edge filtering threshold is set to 30, that is, the shortest side of the minimum circumscribed rectangular border of the defect is detected. If it is less than 30, it is filtered.

[0121] The gradient filtering threshold is set to 40, that is, based on the detected defect area, the gradient value of each pixel in the area is calculated, and it is determined whether the maximum gradient is less than 40. If so, it is determined to be a shallow scratch and needs to be filtered.

[0122] Specifically, the third acquisition device 104 is used to acquire a back low-exposure image and a back high-exposure image of the product after the product is grabbed.

[0123] As an example, the third acquisition device 104 may optionally include four bar light sources and an industrial camera, and the overall imaging may be a bright field or a dark field by controlling the number of bar light sources that are turned on.

[0124] As an example, assume that after the robot grabs the product, the industrial camera collects a 4096*3000 image. The image contains four products, which can be segmented by ROI area. The coordinates of the upper left corner and lower right corner of the ROI area corresponding to each suction cup in the image are as follows:

[0125] Suction cup ROI: [[12,351],[1902,1635]]

[0126] Suction cup No. 2 ROI: [[2175,412],[4074,1719]]

[0127] Suction cup No. 3 ROI: [[20,1482],[1896,2707]]

[0128] Suction cup No. 4 ROI: [[2130,1530],[4029,2782]]

[0129] Specifically, the control device 106 is also used to obtain the number of holes on the surface of the product based on the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image and the back high-exposure cut-out image, so as to judge whether the product is qualified based on the scratch defects, dirt defects and the number of holes on the front and back of the product, and classify and store the products according to the judgment results.

[0130] In a possible implementation, the control device 106 also includes a first defect recognition unit and / or a second defect recognition unit, which is also used to obtain the number of holes on the surface of the product based on the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image and the back high-exposure cut-out image; an area division unit, which is used to divide the front and back of the product into functional areas and non-functional areas; a judgment unit, which is used to judge whether the number of defects contained in the functional areas and non-functional areas is greater than the corresponding number threshold and whether the number of holes on the product surface is not equal to the hole number threshold, if so, the product is an unqualified product, if not, the product is a qualified product; a classification unit, which is used to grab unqualified products and place them in the first storage area, and grab qualified products and place them in the second storage area.

[0131] As an example, the area division unit may be selected according to the following Figure 6 The area division template shown divides the front and back of the product into functional areas and non-functional areas, where the black area is the functional area and the white area is the non-functional area. It is worth noting that since the robot will rotate a certain angle when grabbing the product, the template also rotates with the rotation angle of the product.

[0132] As an example, the quantity threshold corresponding to the functional area is 0, the quantity threshold corresponding to the non-functional area is 3, and the quantity threshold corresponding to the number of holes is arbitrarily set by the product model or customer requirements. That is, if there is one defect in the functional area, or there are three or more defects in the non-functional area, or the number of holes is greater than the set quantity threshold, then the corresponding product is unqualified, otherwise it is qualified. When the minimum circumscribed rectangle of each type of defect intersects with the black area, it can be considered that the defect is located in the functional area.

[0133] As an example, after receiving the judgment result of the judgment unit, the classification unit can grab unqualified products and place them in the first storage area, and grab qualified products and place them in the second storage area, wherein the first storage area and the second storage area are arbitrarily set by technical personnel in this field.

[0134] In this embodiment, a product defect visual inspection system of the present invention is provided, comprising: a transport device; a first acquisition device, a second acquisition device and a third acquisition device arranged along the travel direction of the transport device, as well as a gripping device and a control device; during product transportation, the first acquisition device acquires a low-exposure image of the front side of the product, the second acquisition device acquires a high-exposure image of the front side of the product, the control device determines the gripping posture according to the low-exposure image of the front side and / or the high-exposure image of the front side to control the gripping device to grip the product, the third acquisition device acquires a low-exposure image and a high-exposure image of the back side of the product after the product is gripped, the control device then identifies the front scratch defect according to the low-exposure image of the front side and identifies the front dirt defect according to the high-exposure image of the front side; identifies the back dirt defect according to the low-exposure image of the back side and identifies the back scratch defect according to the high-exposure image of the back side, thereby improving the detection technology, reducing false detection and missed detection caused by subjective judgment, improving the accuracy and consistency of the detection results, and being able to identify a variety of defect types at the same time to adapt to the complex and changeable defect characteristics of the product, and optimizing the structure and process of the detection system to meet the needs of large-scale production, improve production efficiency, and improve the detection accuracy of small-size defects. It solves the problems of low accuracy and consistency of detection results of existing monitoring systems, inability to adapt to complex and changeable defect characteristics of products, and low detection accuracy for small-size defects.

[0135] On the other hand, Figure 7 As shown, the present invention also provides a product defect visual detection method, comprising:

[0136] S1: Collect low-exposure images and high-exposure images of the front of the product;

[0137] Specifically, it is optional but not limited to moving the product in sequence along the direction of travel of the conveyor, and sequentially arranging a plurality of acquisition devices in sequence along the direction of travel of the conveyor to acquire a low-exposure image of the front side of the product and a high-exposure image of the front side of the product;

[0138] S2: determining a grasping posture according to the front low-exposure image or / and the front high-exposure image to control a grasping device to grasp the product;

[0139] Specifically, it is optional but not limited to obtaining the product outline in the front low-exposure image and / or the front high-exposure image to obtain the minimum circumscribed rectangle of the product outline, and then obtaining the product size data based on the vertex or center point data of the minimum circumscribed rectangle to determine the grasping posture corresponding to each product, thereby controlling the grasping device to grasp the product according to the grasping posture.

[0140] Preferably, the grasping posture is determined according to the front low-exposure image and / or the front high-exposure image to control the grasping device. The specific steps of grasping the product may optionally include:

[0141] S21: Build and train a neural network model that takes low-exposure images or high-exposure images as input and product outlines as output, which is an instance segmentation model;

[0142] Specifically, the neural network model can use any image recognition model in the prior art to construct an initial neural network model; then batch collect low-exposure images or high-exposure images of different workpieces of the same model for annotation to construct a training data set: then input the initial neural network model according to the training data set to train the various parameters of the initial neural network model to obtain a trained neural network model. It is worth noting that this step is a preparatory work, which can be constructed and trained in advance, or it can be carried out simultaneously with subsequent steps, as long as it is constructed and trained before the neural network model is needed to segment the product contour.

[0143] For example, the mask-_rcnn_R_50_FPN_3x model can be used as the initial neural network model. A number of low-exposure images and / or high-exposure images are collected in advance, and the product contours in each image are annotated to construct a training set. The model training parameters are then set to input the annotated low-exposure images and / or high-exposure images into the initial neural network model. The initial neural network model is trained according to the set model training parameters to obtain a trained neural network model. The model training parameters can be optionally set as:

[0144] Number of classes (num_classes): 1

[0145] Class name (classes_name): ['bg']

[0146] Batch size (batch_size): 32

[0147] Learning rate: 0.0025

[0148] Epochs: 20000

[0149] Save interval (save_period): 5000

[0150] Maximum size (max_size): 512

[0151] Minimum size (min_size): [512,]

[0152] Among them, bg represents the product, the maximum size is to scale the longest side of the input image to 512, and the minimum size is to scale the shortest side proportionally according to the scaling ratio of the longest side.

[0153] S22: Input the low-exposure image of the front of the product and the high-exposure image of the front of the product into the instance segmentation model to obtain the contours of the products in the image, and obtain the minimum bounding rectangle of each product contour, so as to obtain product size data according to the minimum bounding rectangle of the product contour; the product size data includes product width and height, product angle, product center point, and product vertex;

[0154] S23: Determine a grasping posture according to the product size data, so as to control the grasping device to grasp the product according to the grasping posture.

[0155] Specifically, the low-exposure image of the front of the product and the high-exposure image of the front of the product collected in step S1 can be optionally input into the instance segmentation model trained in step S21 to obtain the product contour in the image, and obtain the minimum circumscribed rectangle of each product contour, so as to obtain the product size data according to the minimum circumscribed rectangle of the product contour, and then determine the grasping posture according to the product size data, so as to control the grasping device to grasp the product according to the grasping posture; the product size data includes product width and height, product angle, product center point, and product vertex.

[0156] For example, the cv::minAreaRect() function in the opencv library can be used to input the product outline into the cv::minAreaRect() function to obtain the minimum enclosing rectangle of the corresponding product outline, as well as product size data such as product width and height, product angle, product center point, and product vertex. More specifically, the product angle can be selected as the angle between the longest side or midline of the minimum enclosing rectangle of the product outline and an arbitrarily set horizontal line. The specific calculation method can be arbitrarily set by those skilled in the art.

[0157] S3: After the product is captured, a low-exposure image of the back side and a high-exposure image of the back side of the product are collected;

[0158] S4: Identify front scratch defects based on the front low-exposure image, and identify front dirt defects based on the front high-exposure image; identify back dirt defects based on the back low-exposure image, and identify back scratch defects based on the back high-exposure image.

[0159] Specifically, after the product is grabbed, it is optional to change the lighting conditions to capture a low-exposure image and a high-exposure image of the back of the product, and then identify the front scratch defect based on the front low-exposure image and the front dirt defect based on the front high-exposure image; identify the back dirt defect based on the back low-exposure image and the back scratch defect based on the back high-exposure image.

[0160] Preferably, the steps of identifying front scratch defects according to the front low-exposure image and identifying front dirt defects according to the front high-exposure image; identifying back dirt defects according to the back low-exposure image and identifying back scratch defects according to the back high-exposure image include:

[0161] S41: construct and train a neural network model that takes the low-exposure cut image as input and the defect position and hole number as output, which is the first defect recognition model;

[0162] For example, the mask_rcnn_R_50_FPN_3x model can be used as the initial neural network model. A number of low-exposure cut-image images are acquired in advance, and the positions of holes and defects in each low-exposure cut-image are marked to construct a training set. Then, the model training parameters are set to input the marked low-exposure cut-image images into the initial neural network model. The initial neural network model is trained according to the set model training parameters to obtain a trained neural network model. The model training parameters can be optionally set as:

[0163] Number of classes (num_classes): 4

[0164] Class names (classes_name): ['bg', 'hole', 'quexain1']

[0165] Batch size (batch_size): 32

[0166] Learning rate: 0.0025

[0167] Epochs: 20000

[0168] Save interval (save_period): 5000

[0169] Maximum size (max_size): 640

[0170] Minimum size (min_size): [640,]

[0171] Among them, bg represents the product, hole represents the hole, quexain1 represents the defect position in the low-exposure cut image, the maximum size is to scale the longest side of the input image to 640, and the minimum size is to scale the shortest side proportionally according to the scaling ratio of the longest side.

[0172] S42: construct and train a neural network model that takes the high-exposure cut image as input and the defect position and hole quantity as output, which is a second defect recognition model;

[0173] For example, the mask_rcnn_R_50_FPN_3x model can be used as the initial neural network model. A number of high-exposure cut-image images are acquired in advance, and the holes and defect positions in each high-exposure cut-image are marked to construct a training set. Then, the model training parameters are set to input the marked high-exposure cut-image images into the initial neural network model. The initial neural network model is trained according to the set model training parameters to obtain a trained neural network model. The model training parameters can be optionally set as:

[0174] Number of classes (num_classes): 4

[0175] Class names (classes_name): ['bg', 'hole', 'zangwu', 'quexain2']

[0176] Batch size (batch_size): 32

[0177] Learning rate: 0.0025

[0178] Epochs: 20000

[0179] Save interval (save_period): 5000

[0180] Maximum size (max_size): 640

[0181] Minimum size (min_size): [640,]

[0182] Among them, bg represents the product, hole represents the hole, quexain2 represents the defect position in the high-exposure cut image, the maximum size is to scale the longest side of the input image to 640, and the minimum size is to scale the shortest side proportionally according to the scaling ratio of the longest side.

[0183] S43: Input the low-exposure image of the back of the product and the high-exposure image of the back of the product into the instance segmentation model to obtain the contours of the products in the image, and obtain the minimum circumscribed rectangle of each product contour;

[0184] S44: Expanding the pixels of the minimum bounding rectangles of the products in the front low-exposure image, the front high-exposure image, the back low-exposure image, and the back high-exposure image to obtain the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image corresponding to the minimum bounding rectangles of the products;

[0185] Specifically, the front low-exposure image, front high-exposure image, back low-exposure image and back high-exposure image obtained in step S1 and step S3 can be optionally input into the instance segmentation model to obtain the product contours in the image, and the minimum bounding rectangle of each product contour is obtained, and then the minimum bounding rectangle of the product is expanded in pixels to obtain the front low-exposure cut image, front high-exposure cut image, back low-exposure cut image and back high-exposure cut image corresponding to the minimum bounding rectangle of each product.

[0186] Preferably, when expanding the pixels of the minimum bounding rectangle of each product, you can optionally obtain an area with 256 pixels extending in all directions with each minimum bounding rectangle as the center, and extract the area from the low-exposure image or the high-exposure image, so as to obtain the low-exposure cut-out image or the high-exposure cut-out image corresponding to the minimum bounding rectangle of each product.

[0187] S45: inputting the front low-exposure cut-out image and the back low-exposure cut-out image into a first defect recognition model to obtain a front scratch defect and a back dirt defect;

[0188] Specifically, all low-exposure cut-out images can be optionally input into the first defect recognition model to obtain the defect position in the image. Since the front low-exposure cut-out image is taken in a dark room without interference from ambient light, the overall image is a dark field, which is darker, while the defect position image is white, which is brighter, thereby obtaining a front scratch defect; the back is taken under ambient light and there is ambient light interference, so the defect detection category of the bright and dark fields is exactly opposite to that of the front detection, thereby obtaining a back dirt defect.

[0189] S46: Inputting the front high-exposure cut-out image and the back high-exposure cut-out image into a second defect recognition model to obtain a front dirt defect and a back scratch defect.

[0190] Specifically, all high-exposure cut-out images can be optionally input into the second defect recognition model to obtain the defect position in the image. Since the low-exposure cut-out image of the front side is taken in a dark room without interference from ambient light, the overall image is a bright field, which is brighter, while the image of the defect position is black, which is darker, thereby obtaining a front dirt defect; the back side is taken under ambient light and there is ambient light interference, so the defect detection categories of the bright and dark fields are exactly opposite to those of the front side detection, thereby obtaining a back scratch defect.

[0191] Preferably, after obtaining the scratch defects and dirt defects on the front and back of the product, it is necessary to judge whether each product is qualified according to the number of defects. Therefore, the method further includes:

[0192] S5: Obtain the number of holes on the product surface based on the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image, so as to judge whether the product is qualified based on the scratch defects, dirt defects, and the number of holes on the front and back of the product, and classify and store the products according to the judgment results.

[0193] Specifically, it is optional but not limited to setting a defect number threshold and a hole number threshold, obtaining the number of holes on the product surface according to the first defect recognition model and the second defect recognition model of steps S41 and S42, and then judging whether the defects on the product surface are greater than the defect number threshold and whether the number of holes is not equal to the hole number threshold. If so, the product is unqualified, if not, the product is qualified, and finally the qualified products and unqualified products are classified and stored according to the judgment results.

[0194] Preferably, the specific steps of S5 may optionally include:

[0195] S51: inputting the front low-exposure cut-out image and the back low-exposure cut-out image into a first defect recognition model, and inputting the front high-exposure cut-out image and the back high-exposure cut-out image into a second defect recognition model to obtain the number of holes on the product surface;

[0196] S52: Divide the product surface into functional areas and non-functional areas;

[0197] Specifically, optional but not limited to Figure 6 The area division template shown divides the front and back of the product into functional areas and non-functional areas, where the black area is the functional area and the white area is the non-functional area. It is worth noting that since the robot will rotate a certain angle when grabbing the product, the template also rotates with the rotation angle of the product.

[0198] S53: Determine whether the number of holes on the product surface is not equal to the hole number threshold. If so, it is an unqualified product. If not, then:

[0199] S54: Determine whether the number of defects contained in the functional area is greater than the first defect number threshold. If so, the product is a non-conforming product. If not, then:

[0200] S55: Determine whether the number of defects contained in the non-functional area is greater than a second defect number threshold. If so, the product is an unqualified product; if not, the product is a qualified product.

[0201] Specifically, the hole number threshold, the first defect number threshold and the second defect number threshold are arbitrarily set by those skilled in the art. Preferably, the first defect number threshold is 0 and the second defect number threshold is 3, that is, no defects may appear in the functional area and no more than 3 defects may appear in the non-functional area, otherwise the product will be unqualified.

[0202] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A product defect visual inspection system, characterized in that: include: Transport devices; A first collecting device, a second collecting device and a third collecting device, as well as a grasping device and a control device, are arranged along the traveling direction of the transport device; During product transportation: The first acquisition device is used to acquire a low-exposure image of the front side of the product; The second acquisition device is used to acquire a high-exposure image of the front side of the product; A control device, used to determine a grasping posture according to the front low-exposure image and / or the front high-exposure image, so as to control the grasping device to grasp the product; A third acquisition device is used to acquire a low-exposure image and a high-exposure image of the back side of the product after the product is grabbed; The control device is also used to identify front scratch defects based on the front low-exposure image and identify front dirt defects based on the front high-exposure image; identify back dirt defects based on the back low-exposure image and identify back scratch defects based on the back high-exposure image.

2. The system according to claim 1, characterized in that Control device, including: An image processing unit, used to perform instance segmentation on the image to obtain product contours, and obtain the minimum circumscribed rectangle of each product contour, so as to obtain product size data according to the minimum circumscribed rectangle of the product contour; the product size data includes product width and height, product angle, product center point, and product vertex; A posture calculation unit, used to determine the grasping posture according to the product size data; The gripping control unit is used to control the gripping device to grip the product according to the gripping posture.

3. The system according to claim 2, characterized in that Image processing unit, including: A segmentation component for performing instance segmentation on the image to obtain product contours; A dimension data calculation component is used to obtain the minimum circumscribed rectangle of each product outline, so as to obtain product dimension data according to the minimum circumscribed rectangle of the product outline; A filter component is used to filter the minimum bounding rectangle of products whose edges are blocked according to a prefabricated edge mask template.

4. The system according to claim 2, characterized in that The control device further comprises: A pixel expansion unit is used to expand the pixels of the minimum circumscribed rectangle of the product in the front low-exposure image, the front high-exposure image, the back low-exposure image, and the back high-exposure image, so as to obtain the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image corresponding to the minimum circumscribed rectangle of each product; A first defect recognition unit, configured to recognize a front scratch defect based on the front low-exposure cut-out image, or recognize a back dirt defect based on the back low-exposure cut-out image; The second defect recognition unit is used to recognize the front dirt defect based on the front high-exposure cut-out image, or to recognize the back scratch defect based on the back high-exposure cut-out image.

5. The system according to claim 4, characterized in that The system further comprises: The control device is also used to obtain the number of holes on the surface of the product based on the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image and the back high-exposure cut-out image, so as to judge whether the product is qualified based on the scratch defects, dirt defects and the number of holes on the front and back of the product, and classify and store the products according to the judgment results.

6. The system according to claim 5, characterized in that The control device further comprises: The first defect recognition unit or / and the second defect recognition unit are further used to obtain the number of holes on the surface of the product according to the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image; Area division unit, used to divide the front and back of the product into functional areas and non-functional areas; A judging unit, used to judge whether the number of defects contained in the functional area and the non-functional area is greater than a corresponding number threshold and whether the number of holes on the surface of the product is not equal to the hole number threshold, if so, the product is an unqualified product, if not, the product is a qualified product; The classification unit is used to grab unqualified products and place them in the first storage area, and grab qualified products and place them in the second storage area.

7. A method for visually detecting product defects, characterized in that: A product defect visual inspection system as claimed in any one of claims 1 to 6, comprising: Collect low-exposure images of the front of the product and high-exposure images of the front of the product; Determine a grasping posture according to the front low-exposure image and / or the front high-exposure image to control a grasping device to grasp the product; After the product is captured, a low-exposure image of the back side and a high-exposure image of the back side of the product are collected; The front scratch defect is identified based on the front low-exposure image, and the front dirt defect is identified based on the front high-exposure image; the back dirt defect is identified based on the back low-exposure image, and the back scratch defect is identified based on the back high-exposure image.

8. The method according to claim 7, characterized in that The steps of determining a grasping posture according to the front low-exposure image and / or the front high-exposure image to control a grasping device to grasp the product include: Build and train a neural network model that takes low-exposure images or high-exposure images as input and product outlines as output, which is an instance segmentation model; Input the low-exposure image of the front of the product and the high-exposure image of the front of the product into the instance segmentation model to obtain the product contour in the image, and obtain the minimum bounding rectangle of each product contour, so as to obtain the product size data according to the minimum bounding rectangle of the product contour; the product size data includes the product width and height, product angle, product center point, and product vertex; The grasping posture is determined according to the product size data, so as to control the grasping device to grasp the product according to the grasping posture.

9. The method according to claim 8, characterized in that Identify front scratch defects based on the front low-exposure image, and identify front dirt defects based on the front high-exposure image; The steps of identifying a back side dirt defect according to a back side low-exposure image and identifying a back side scratch defect according to a back side high-exposure image include: Construct and train a neural network model that takes low-exposure cut-out images as input and outputs defect locations and hole numbers as the first defect recognition model; Construct and train a neural network model that takes high-exposure cut-out images as input and outputs defect locations and hole numbers as the second defect recognition model; Input the low-exposure image of the back of the product and the high-exposure image of the back of the product into the instance segmentation model to obtain the product outlines in the image and obtain the minimum circumscribed rectangle of each product outline; Expand the pixels of the minimum bounding rectangles of the products in the front low-exposure image, the front high-exposure image, the back low-exposure image, and the back high-exposure image to obtain the front low-exposure cut-out image, the front high-exposure cut-out image, the back low-exposure cut-out image, and the back high-exposure cut-out image corresponding to the minimum bounding rectangles of each product; Inputting the front low-exposure cut-out image and the back low-exposure cut-out image into a first defect recognition model to obtain a front scratch defect and a back dirt defect; The front high-exposure cut-out image and the back high-exposure cut-out image are input into the second defect recognition model to obtain the front dirt defect and the back scratch defect.

10. The method according to claim 8, characterized in that The method further comprises: Input the front low-exposure cut-out image and the back low-exposure cut-out image into the first defect recognition model, and input the front high-exposure cut-out image and the back high-exposure cut-out image into the second defect recognition model to obtain the number of holes on the product surface; Divide the product surface into functional and non-functional areas; Determine whether the number of holes on the product surface is not equal to the hole number threshold. If so, it is an unqualified product. If not, then: Determine whether the number of defects contained in the functional area is greater than the first defect number threshold. If so, the product is a non-conforming product. If not, then: It is determined whether the number of defects contained in the non-functional area is greater than a second defect number threshold value. If so, the product is an unqualified product; if not, the product is a qualified product.

Images