A simple defect detection method and system based on object recognition

Abstract

The present invention belongs to the field of image recognition and discloses a simple defect detection method based on object recognition, comprising the following steps: Step 1: Obtaining a total image through a camera; the total image contains an object to be identified; Step 2: Using an AI algorithm to train and identify the type and boundary of the object in the total image, and separating the object image from the total image according to the boundary; Step 3: Processing the object image through a semantic segmentation algorithm to obtain the object's outline; Step 4: Rotating the object image according to the object's outline so that the angle of the object in the image is arranged according to a preset angle; Step 5: Based on the characteristics of each component on the object, identifying the component containing the characteristic from the object image to obtain a detection result. This method combines traditional computer graphics with AI algorithms, fully leveraging the advantages of both methods while complementing their respective shortcomings, and rapidly obtaining defect detection results with minimal computing power.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a simple defect detection method and system based on object recognition. Background Art

[0002] Currently, there are two main ways to perform defect detection through images.

[0003] Computer Graphics

[0004] Through common computer graphics operations such as grayscale, expansion, scaling, cropping, and deformation, features on the surface of the object are obtained, and defects are judged based on the features.

[0005] There are the following deficiencies and shortcomings:

[0006] 1. It is impossible to determine the boundaries of the entire object.

[0007] If the boundaries of the object cannot be determined, it is impossible to determine whether all features can be captured, and it is impossible to locate where image operations and defect detection are required on the image.

[0008] If the fixture is used, the cost is high and the operation time is increased. It is also not suitable for simultaneous inspection of batches of objects.

[0009] Object location is performed by labeling and finding features. This is not easy to use, as many devices do not have labels. Furthermore, if two objects are very similar, it is impossible to distinguish them effectively.

[0010] 2. Unable to distinguish what the object is, it can only be judged by simple shape or color.

[0011] If other objects that are somewhat similar accidentally enter the camera, it will be misjudged.

[0012] 3. Because it can't distinguish specific object categories, it can't automatically associate results with objects and requires manual selection. This is not conducive to systems that reuse different types of objects. For example, object 1 needs to be inspected for defects in the USB port, object 2 needs to be inspected for soldering problems on the LED light, and object 3 needs to be inspected for antenna installation. In these cases, it's necessary to select the object category before testing.

[0013] However, the advantage of computer graphics is that it does not require the computing power of the operating environment.

[0014] 4. The offset angle of the object cannot be determined. If the rotation angle is large, all reference coordinates will be invalid.

[0015] AI algorithm recognition

[0016] The use of AI to detect object defects is gaining popularity, effectively improving the shortcomings of graphics.

[0017] There are the following deficiencies and shortcomings:

[0018] 1. There are certain computing power requirements for the operating environment

[0019] Currently, there are many classification and image recognition algorithms, all of which require operations such as convolution, pooling, and activation. Algorithms with higher accuracy require higher computing power, otherwise real-time detection is impossible.

[0020] 2. Image input size has requirements

[0021] AI algorithms have specific input size requirements, which are typically determined before training. While overly large inputs can detect tiny objects, they can take a long time to train and infer, requiring high-performance computing devices to run in real time.

[0022] While performance improves when the image size is too small, it still struggles with smaller objects. Currently, the resolution ranges from 200x200 to 600x600. Even with a resolution of 600x600, surface defects will also be reduced when the image is scaled down. This presents significant challenges when identifying subtle defects.

[0023] 3. The object to be identified needs to be determined first

[0024] For example, LED lights come in all shapes and sizes, and have various soldering irons around them. This requires a significant amount of effort when labeling the dataset, as we only keep the LED components and exclude other components.

[0025] If the LED is replaced with a device of a different shape, retraining is required to adapt.

[0026] Training also requires introducing different defect types. For example, for welding problems, there are cold joints, no joints, and welding errors. Each category must be photographed for training. Furthermore, solder joints of different shapes must be trained repeatedly. This consumes a significant amount of time, including photographing, labeling, and other data collection tasks.

[0027] This algorithm is based on the training of AI algorithms, and a lot of training is required to calculate accurate information.

[0028] For example, CN201811634845.0 discloses an AI algorithm for car labeling and recognition based on two-dimensional labels. The algorithm constructs two-dimensional car labels, which include first-dimensional car brand labels and second-dimensional car parts labels; constructs a car brand label set; and uses a model generated after training car brand labeling based on two-dimensional labels. When using the model to identify car brands in images in movies and TV shows, the original recognition results include the car brand and similarity in the one-dimensional label, as well as the car parts similarity value in the two-dimensional label. Based on the similarity in the two dimensions, the algorithm recalculates the similarity with the car brand using a weighted fitting algorithm, thereby determining the car brand and car area appearing in each frame of the video. This algorithm effectively solves the problem that conventional AI methods cannot effectively identify car brands in long shots in movies and TV shows, or when there is occlusion.

[0029] The problem with simply using AI algorithms is that once it involves matching parts, the training set becomes very large and the amount of calculation is also very large. In addition, there are high requirements for the standardization of images.

[0030] The technical problem that this solution aims to solve is: how to use less computing power to quickly identify common defects that occur during the generation process. Summary of the Invention

[0031] The purpose of the present invention is to provide a simple defect detection method based on object recognition. This method combines traditional computer graphics and AI algorithms to give full play to the advantages of the two methods, complement their respective shortcomings, and quickly obtain defect detection results with less computing power.

[0032] This invention focuses on the most important requirements in practical application scenarios, such as whether indicator lights are working properly, antennas are installed, connectors are soldered, screws are tightened, and other common inspection scenarios. Minor defects, such as surface scratches and component soldering issues, are not considered in this system design. This ensures maximum defect coverage at the lowest cost.

[0033] At the same time, this system design is also compatible with a variety of models and objects. It does not require manual intervention for category selection. After automatically deriving the category through the AI ​​algorithm, the corresponding algorithm plug-in is selected for screening to improve the degree of automation.

[0034] Finally, this system design supports simultaneous detection of multiple types of items, improving the overall detection efficiency.

[0035] To achieve the above objectives, the present invention provides the following technical solution: a simple defect detection method based on object recognition, comprising the following steps:

[0036] Step 1: Obtaining a total image through a camera; the total image contains an object to be identified;

[0037] Step 2: Use AI algorithm training to identify the type and boundaries of objects in the total image, and separate the object image from the total image according to the boundaries;

[0038] Step 3: Process the object image through the semantic segmentation algorithm to obtain the object's outline;

[0039] Step 4: Rotate the image of the object according to the outline of the object so that the angle of the object in the image is arranged according to the preset angle;

[0040] Step 5: Based on the features of each component on the object, the parts containing the features are identified from the image of the object to obtain the detection results.

[0041] In the above-mentioned simple defect detection method based on object recognition, in step 2, the specific method of identifying the type of object is: through the object recognition algorithm, pictures of various types of objects are taken in advance for training, and the trained model is used for customized object recognition.

[0042] In the above-mentioned simple defect detection method based on object recognition, the step 3 is specifically as follows:

[0043] Step 31: Classify the pixels in the object image through semantic segmentation to obtain the filled color blocks of each object in the image;

[0044] Step 32: Draw the outline of the object based on the filled color block.

[0045] In the above-mentioned simple defect detection method based on object recognition, the preset angle in step 4 refers to the angle at which the object is in a horizontal state.

[0046] In the above-mentioned simple defect detection method based on object recognition, step 4 further includes an integrity identification step of identifying the integrity of the object, and step 5 is performed after the integrity identification step is passed;

[0047] The integrity identification step is: calculating the size of the object based on the outline of the object, comparing the size of the object with the preset size, if it matches, the comparison passes, if it does not match, the comparison fails; the preset size is the actual size of the object.

[0048] In the above-mentioned simple defect detection method based on object recognition, in step 5, the component is one or more of an interface, a lamp, and a label.

[0049] In the above-mentioned simple defect detection method based on object recognition, during the identification process of the interface, if the color of the interface is different from that of other main positions of the object, the interface is identified based on the color feature.

[0050] In the above-mentioned simple defect detection method based on object recognition, during the interface recognition process, if the color of the interface is different from that of other main positions of the object, the interface is recognized based on the color characteristics and the size of the area where the color is located.

[0051] In the above-mentioned simple defect detection method based on object recognition, during the lamp recognition process, the lamp is in a lighting state, and the lamp is recognized based on the brightness characteristics of the lamp.

[0052] Finally, the present invention also discloses a simple defect detection system based on object recognition, including a camera, an AI recognition module, and an image recognition module;

[0053] The camera is used to obtain a total image; the total image contains an object to be identified;

[0054] AI recognition module: used to obtain the overall image from the camera, identify the type and boundaries of the object in the overall image, separate the object image from the overall image according to the boundary, process the object image through the semantic segmentation algorithm, obtain the object's outline, and adjust the image angle;

[0055] Image recognition module: used to identify the parts containing the features from the image of the object based on the features of each component on the object to obtain the detection results.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The outstanding effect of the present invention is that the method of the present invention, by combining traditional computer graphics and AI algorithms, fully utilizes the advantages of the two methods and complements their respective shortcomings. It focuses on the greatest needs of actual application scenarios, such as whether the indicator light is working properly, whether the antenna is installed, whether the interface is welded, whether the screws are tightened, and other general conventional inspection scenarios. Excessively small defects, such as surface scratches and whether the components are welded, are not taken into consideration in the design of this system. In this way, the maximum proportion of defects can be covered at the lowest cost.

[0058] At the same time, this system is designed to be compatible with a variety of models and objects. No manual intervention is required for category selection. After the AI ​​algorithm automatically determines the category, the corresponding algorithm plug-in is selected for screening, improving the degree of automation.

[0059] Finally, this system design supports simultaneous detection of multiple types of items, improving the overall detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a general image of a practical case of Example 1 of the present invention;

[0061] Figure 2 and 3This is a case of semantic recognition in embodiment 1 of the present invention;

[0062] Figure 4 This is a case of adjusting the angle of an image of an object according to embodiment 1 of the present invention;

[0063] Figure 5 This is a flowchart of the method of embodiment 1 of the present invention;

[0064] Figure 6 This is a structural block diagram of the system of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] Example 1

[0067] refer to Figure 5 , a simple defect detection method based on object recognition, comprising the following steps:

[0068] Step 1: Obtaining a total image through a camera; the total image contains an object to be identified;

[0069] by Figure 1 For example, Figure 1 There are 3 devices and 2 PCB boards in the file, and the devices and PCB boards are objects.

[0070] During the production process, we usually place cameras above the production line to capture the overall image. The camera specifications are not too high, generally speaking, the camera maintains a low pixel, such as 1920x1080 or 1024x720 pixels.

[0071] Step 2: Use AI algorithm training to identify the type and boundaries of objects in the total image, and separate the object image from the total image according to the boundaries;

[0072] In the prior art, there are many AI algorithms. Since the AI ​​algorithm of the present invention does not involve the identification of parts, it is mainly used for real-time detection.

[0073] Currently, open-source algorithms like SSD-MobileNetV2-lite and YoloV4-Tiny offer excellent performance and are suitable for embedded systems. For example, YoloV4-Tiny achieves real-time performance of 36 fps on the Jetson Nano and 50 fps on the MX8Plus, exceeding real-time requirements. It can even simultaneously inspect four production lines, each at approximately 9 to 12 fps, meeting real-time requirements.

[0074] The relevant website https: / / www.cnblogs.com / zyly / p / 9180485.html records the specific methods of sliding window and Bounding Box prediction, and uses the convolution of the AI ​​algorithm-sliding window method to obtain a more accurate bounding box.

[0075] This step satisfies the real-time detection requirement and is of great significance for improving detection efficiency.

[0076] refer to Figure 1 , the boundary is as Figure 1 The box in the image only defines a vague range of the object and is mainly used for subsequent semantic analysis to perform contour analysis and category recognition.

[0077] Step 3: Process the object image through the semantic segmentation algorithm to obtain the object's outline;

[0078] Semantic segmentation algorithms use pixel sharpness and transitions to identify different contours, and then use training data to determine whether these contours belong to the same object. They primarily distinguish between color blocks and boundary lines.

[0079] Semantic segmentation algorithm can be referenced:

[0080] https: / / blog.csdn.net / qq_31820761 / article / details / 82628873, which discloses a specific processing method for a semantic segmentation algorithm using a bounding box as auxiliary information and another specific processing method for a semantic segmentation algorithm using a scribbled line as auxiliary information.

[0081] Specifically:

[0082] Step 31: Classify the pixels in the object image through semantic segmentation to obtain the filled color blocks of each object in the image;

[0083] by Figure 2 and 3 For example, Figure 2 is the total image containing the object, Figure 3is the filled color block obtained through semantic segmentation.

[0084] Each pixel is classified to accurately determine the outline of an object. This is because each pixel must be classified, such as whether it belongs to a car, a person, or the background. There are no objects, only pixels. Once the pixel set is obtained, an algorithm can be used to calculate the object's rotation angle, ultimately rotating it to the horizontal angle.

[0085] Step 32: Draw the outline of the object based on the filled color block. Figure 4 , Figure 4 An example of obtaining the outline of an object by performing semantic segmentation according to step 3 is shown.

[0086] Step 4: Rotate the angle of the object's image according to the object's outline so that the object in the image is rotated to the horizontal;

[0087] In order to accurately carry out step 5, it is necessary to eliminate problems that occur during the shooting process, such as only half of the object is photographed. What should be done?

[0088] In view of this situation, before step 5, an integrity identification step of identifying the integrity of the object is also included, which is specifically as follows:

[0089] The size of the object is calculated based on its outline and compared with the preset size. If they match, the comparison passes; if not, the comparison fails. The preset size is the actual size of the object.

[0090] Specifically, one problem with AI algorithm output is that it doesn't tell you whether the entire object is captured. The algorithm identifies an object based on the matching of certain features, but it can't tell if parts of the image are missing.

[0091] Therefore, before sending the image to the next-level graphics algorithm, an image integrity check must be performed. Because this system is used on a production line, the camera shoots vertically downward from above, capturing only a single surface of the object. The check is then performed based on the aspect ratio of this surface.

[0092] For example, the dev1 and dev2 devices in the figure above have completely different aspect ratios. Assuming the aspect ratio of the Dev1 surface to be inspected is 2.3:1, the Dev1 boundary coordinates obtained by the AI ​​algorithm are used to determine whether the aspect ratio of the bounding box is within the 2.3:1 range.

[0093] Step 5: Based on the features of each component on the object, the parts containing the features are identified from the image of the object to obtain the detection results.

[0094] The components are one or more of an interface, a lamp, and a label.

[0095] by Figure 1 For example, each type of device or PCB has its own unique characteristics and interfaces. Therefore, different algorithms need to be designed based on the identified type. These algorithms primarily focus on checking antenna and screw installation, proper LED operation, soldering of various interfaces, and correct label printing. Being able to complete these inspection requirements can generally meet over 90% of the requirements.

[0096] Specifically, if during the identification process of the interface, the color of the interface is different from that of other main positions of the object, the interface is identified based on the color feature.

[0097] The power interface and the cable interface both have green shells. You can filter the green to see whether the outlines of the two interfaces exist.

[0098] The USB interface will have silver and white features, and the network port will have a network port light feature, which is also achieved through color filtering.

[0099] In a specific case, the network port light is green, and the power and plug-in ports are also green. In order to distinguish the two during the graphic recognition process, the contour detection function is used to return the size of each area to determine the area of ​​the green graphic and make a distinction.

[0100] The OCR (Optical Character Recognition) algorithm is quite unique in the label recognition process. Both open-source and paid algorithms offer excellent results. The algorithm simply needs to capture the correct label position and feed it into the OCR algorithm. To prevent interference from other printed characters, each device has specific requirements for label placement and orientation. In practice, the algorithm simply determines whether the image is at 0 degrees or 180 degrees. Since there are only two possible angles, the image can be cropped and fed into the OCR algorithm accordingly.

[0101] During the lamp recognition process, the lamp is in a lighting state, and the lamp is recognized based on its brightness characteristics.

[0102] The specific process is:

[0103] Whether it's a complete device or a PCB, the proper soldering and installation of LEDs can be determined by checking whether they light up or go off when powered on. When an LED is on, the brightness difference is significantly greater than when it's off. Color filtering can be used to determine how many lights are functioning properly.

[0104] First, perform color filtering on the original image;

[0105] After color filtering, only the content of the corresponding color will be retained, and other areas will turn black;

[0106] Then grayscale processing and binarization processing are performed to retain the areas with highly consistent colors;

[0107] Finally, after dilation and erosion to remove the capillaries of the image, a relatively complete image can be obtained;

[0108] Finally, the box of the LED light is obtained through the contour detection function. The box mentioned here refers to the coordinate description of the object outline, which is a general term for the coordinates x, y, width, height, etc.

[0109] When all the lamps have gone through this round of processing, it will be possible to determine how many lamps are working and whether there are any defects.

[0110] In summary, each device and PCB type determines which algorithm to use, how many lights are considered qualified, what interfaces are qualified, and even the relative position of the clipping for OCR judgment. Therefore, a data structure design is necessary that can well accommodate and scale to different object types.

[0111] by Figure 1 Taking the specific practical application scenario in as an example, the following code comprehensively judges and outputs the results.

[0112]

[0113]

[0114] As shown in the above structure, the aspect ratio of the object, the required calculations, the interface, LED color and format, and the OCR keywords and coordinates are defined.

[0115] The present invention has the following advantages compared with the prior art based on the tasks implemented:

[0116] 1. For front-end object recognition, a simple AI algorithm can be selected because the overall category characteristics of the production are obvious, and high-power equipment is not required, which reduces deployment costs and improves real-time performance.

[0117] 2. Supports simultaneous detection of multiple, multiple, and multiple types of objects. Through the configuration file of the object category, it determines what kind of graphics algorithm calculation needs to be performed for each type of object to obtain the detection result.

[0118] 3. After object recognition, semantic analysis and contour angle calculation are used to determine the horizontal coordinates of the object. The object's aspect ratio is then used to determine whether the entire object has been captured.

[0119] 4. The graphics algorithm used in the simple system focuses on several commonly used scenarios, such as interfaces, LED lights, and label text recognition. It does not analyze subtle images such as surface scratches, components, or traces, in order to meet the needs of most scenarios.

[0120] 5. By combining AI and computer graphics algorithms, we overcome the shortcomings of single computer graphics algorithms, such as the inability to identify object categories, determine object boundaries, and determine whether all objects are captured. Overcoming AI algorithms requires extensive training and correction of images, time, and computing power, as they require interface, LED light, and OCR detection and judgment.

[0121] 6. By focusing on a simple system for a few scenarios, the requirements for image size and clarity are reduced. While meeting the requirements of most scenarios, the deployment cost is reduced and multi-channel real-time detection is provided to improve efficiency.

[0122] Through the design of this system, we can be closer to most of the actual needs of current factory production, improve detection efficiency, reduce detection costs, and achieve automation and intelligence.

[0123] refer to Figure 6 , Figure 6 A system for implementing the above method is shown, which includes: a camera 1, an AI recognition module 2, and an image recognition module 3;

[0124] The camera is used to obtain a total image; the total image contains an object to be identified;

[0125] AI recognition module: used to obtain the overall image from the camera, identify the type and boundaries of the object in the overall image, separate the object image from the overall image according to the boundary, process the object image through the semantic segmentation algorithm, obtain the object's outline, and adjust the image angle;

[0126] Image recognition module: used to identify the parts containing the features from the image of the object based on the features of each component on the object to obtain the detection results.

[0127] There can be multiple image recognition modules, each of which is responsible for identifying different parts. Figure 1 The device 1 has a light and an interface, so the image of the segmented object can be sent to two independent image recognition modules for recognition, and the recognition results can be obtained and summarized.

[0128] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A simple defect detection method based on object recognition, characterized in that: The steps include: Step 1: Obtaining a total image through a camera; the total image contains an object to be identified; Step 2: Use AI algorithm training to identify the type and boundaries of objects in the total image, and separate the object image from the total image according to the boundaries; Step 3: Process the object image through the semantic segmentation algorithm to obtain the object's outline; Step 4: Rotate the image of the object according to the outline of the object so that the angle of the object in the image is arranged according to the preset angle; Step 5: Based on the features of each component on the object, the parts containing the features are identified from the object image to obtain the detection results; Said step 4 further includes an integrity identification step for identifying the integrity of the object, and step 5 is performed after the integrity identification step is passed; The integrity identification step comprises: calculating the size of the object according to the outline of the object, comparing the size of the object with a preset size, and if they match, the comparison passes; if not, the comparison fails; the preset size is the actual size of the object; In step 5, the component is one or more of an interface, a lamp, and a label; During the identification process of the interface, if the color of the interface is different from that of other main parts of the object, the interface is identified based on the color feature; During the identification process of the interface, if the color of the interface is different from that of other main parts of the object, the interface is identified based on the color characteristics and the size of the area where the color is located; During the lamp identification process, the lamp is in a lit state, and the lamp is identified based on the brightness characteristics of the lamp; In step 2, the specific method of identifying the type of object is: using an object recognition algorithm, pre-taking pictures of various types of objects for training, and using the trained model to perform customized object recognition; The step 3 is specifically as follows: Step 31: Classify the pixels in the object image through semantic segmentation to obtain the filled color blocks of each object in the image; Step 32: Draw the outline of the object based on the filled color block; The preset angle in step 4 refers to the angle at which the object is in a horizontal state.

Images