Vision-based bad part detection method and apparatus

Abstract

The application relates to the technical field of image processing, and discloses a vision-based defective part detection method and device; the method comprises the following steps: collecting a target image of a region of interest, the region of interest comprising a standard region where a to-be-detected part is located; detecting the target image based on a part detection model to determine whether the to-be-detected part is a defective part; wherein the part detection model comprises at least one of the following: a part absence detection model, a part deviation detection model and a part defect detection model; the part absence detection model is used for detecting whether the to-be-detected part is absent based on the color of the to-be-detected part; the part deviation detection model is used for detecting whether the to-be-detected part deviates based on part features of the to-be-detected part; and the part defect detection model is used for detecting whether the to-be-detected part has defects based on a deep learning model. The application can improve the detection efficiency, detection precision and detection time of defective parts.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a vision-based method and apparatus for detecting defective parts. Background Technology

[0002] With the continuous development of the manufacturing industry, ensuring product quality has become one of the key factors in enterprise competition. Among them, the accuracy of the installation position of parts on the middle plate of the product is an important link in the quality control of the product. If there is any misalignment, it may affect the assembly of other parts in subsequent processes. Therefore, in order to ensure product quality and reduce rework between different processes, it is more important to add manual or industrial visual inspection processes to the quality inspection process.

[0003] Currently, the installation of parts on the middle plate in factories generally relies on manual visual inspection or manual measurement to determine whether each material is misaligned or whether the misalignment is within a reasonable range. This method has a series of problems, including limitations in measurement accuracy, inconsistent standards among different personnel, and strong subjectivity leading to missed inspections or misjudgments. Summary of the Invention

[0004] In view of the above situation, this application provides a vision-based method and apparatus for detecting defective parts, which aims to solve the above problems or at least partially solve the above problems.

[0005] In a first aspect, embodiments of this application provide a vision-based method for detecting defective parts, characterized in that the method includes: acquiring a target image of a region of interest, the region of interest including a standard region where the part to be tested is located; detecting the target image based on a part detection model to determine whether the part to be tested is a defective part; wherein, the part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested has a defect based on a deep learning model.

[0006] Secondly, embodiments of this application also provide a vision-based defective parts detection device, the device comprising: an acquisition module for acquiring a target image of a region of interest, the region of interest including a standard area where the part to be tested is located; and a detection module for detecting the target image based on a parts detection model to determine whether the part to be tested is a defective part; wherein the parts detection model includes at least one of the following: a parts missing detection model, a parts offset detection model, and a parts defect detection model; the parts missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the parts offset detection model detects whether the part to be tested is offset based on the parts features of the part to be tested, and the parts defect detection model detects whether the part to be tested has a defect based on a deep learning model.

[0007] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the steps described in the first aspect.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the steps described in the first aspect.

[0009] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0010] By acquiring target images of the standard area where the part to be tested is located, and processing and detecting these images based on a part inspection model, it is possible to determine whether the part to be tested is defective. Since the part inspection model includes a part missing detection model, a part offset detection model, and a part missing detection model, it can detect whether the part to be tested is missing, whether it is offset, or whether it has defects, thus accurately detecting the true condition of the part to be tested. This ensures that suitable precision parts are provided for the product. Furthermore, using the model for part inspection saves manpower, resources, and time, and improves inspection accuracy. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0012] Figure 1 A schematic flowchart of the vision-based defective parts detection method provided in an embodiment of this application is shown;

[0013] Figure 2 A flowchart of a vision-based defective parts detection method according to another embodiment of this application is shown;

[0014] Figure 3 This illustration shows the effect of the RGB conversion LAB channel provided in the embodiment of this application;

[0015] Figure 4 A flowchart of a vision-based defective parts detection method provided in another embodiment of this application is shown;

[0016] Figure 5 A flowchart of a vision-based defective parts detection method provided in another embodiment of this application is shown;

[0017] Figure 6 A schematic diagram of the bilinear interpolation principle provided in an embodiment of this application is shown.

[0018] Figure 7 A flowchart of a vision-based defective parts detection method provided in another embodiment of this application is shown;

[0019] Figure 8 A flowchart of a vision-based defective parts detection method provided in another embodiment of this application is shown;

[0020] Figure 9 A structural diagram of the vision-based defective parts detection device provided in an embodiment of this application is shown;

[0021] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0024] As described in the background section, the installation of parts on the middle plate in factories currently generally relies on manual visual inspection or manual measurement to determine whether each material is misaligned or whether the misalignment is within a reasonable range. This method has a series of problems, including limitations in measurement accuracy, inconsistent standards among different personnel, and strong subjectivity leading to missed inspections or misjudgments.

[0025] Based on this, in order to more stably and efficiently solve the cumbersome operation of existing medium plate parts defect detection and improve the shortcomings of the original medium plate parts defect detection, this application provides a vision-based defective parts detection method. By acquiring the target image of the standard area where the part to be tested is located, the target image is processed and detected based on the part detection model to determine whether the part to be tested is a defective part.

[0026] The present application will now be described in detail through specific embodiments.

[0027] Figure 1 This illustration shows a flowchart of a vision-based defective parts detection method provided in an embodiment of this application. Figure 1 It can be seen that this application includes at least steps S101-S102:

[0028] Step S11: Acquire the target image of the region of interest.

[0029] The region of interest includes the standard area where the part under test is located. The standard area refers to the area where a standard part is located. To avoid errors, the area of ​​the standard area is appropriately enlarged as the region of interest. The enlarged area can be specified by the technician.

[0030] Step S12: Detect the target image based on the part detection model to determine whether the part to be tested is a defective part.

[0031] The part inspection model includes at least one of the following: part missing detection model, part offset detection model, and part defect detection model.

[0032] Specifically, the missing part detection model detects whether the part under test is missing based on its color, the offset detection model detects whether the part under test is offset based on its features, and the defect detection model detects whether the part under test has defects based on a deep learning model.

[0033] It is worth noting that the model mentioned in this application does not necessarily represent a machine learning model, but is merely a general concept of a processing procedure. The specific process of model detection can be a machine learning process or an implementation process of a processing method.

[0034] from Figure 1As shown in the method, this application acquires a target image of the standard area where the part to be tested is located, and processes and detects the target image based on the part detection model to determine whether the part to be tested is a defective part. Since the part detection model includes a part missing detection model, a part offset detection model, and a part missing detection model, it can detect whether the part to be tested is missing, whether the part to be tested is offset, and whether the part to be tested has defects, thereby accurately detecting the true condition of the part to be tested, ensuring that the product is provided with precision parts of suitable quality. At the same time, by using the model to detect the parts, manpower, material resources, and time are saved, and the detection accuracy is improved.

[0035] The following explains how to determine whether the part to be tested is a defective part when the part inspection models are the part missing detection model, the part offset detection model, and the part defect detection model.

[0036] I. The part inspection model is a part missing detection model.

[0037] In some embodiments, the target detection method of the missing part detection model is determined by the color of the part to be tested, and the part to be tested is detected as a defective part based on the target detection method.

[0038] Specifically, if the saturation of the color of the part to be tested and the background color of the target image is greater than a threshold, the part to be tested is classified as category 1, and the target detection method of the part missing detection model is determined to be the first detection method. If the saturation of the color of the part to be tested and the background color of the target image is less than or equal to a threshold, the part to be tested is classified as category 2, and the target detection method of the part missing detection model is determined to be the second detection method.

[0039] For example, if the background color is white and the color of the part to be tested is a distinct color such as yellow or red, like a bright yellow copper sheet or a red rubber sleeve, then the part to be tested can be classified as Category 1, and the corresponding target detection method is the first detection method. If the color of the part to be tested is white, beige, or other colors that are not clearly distinguishable from the background color, then the part to be tested can be classified as Category 2, and the corresponding target detection method is the second detection method.

[0040] In this embodiment, parts can be pre-classified according to their colors. When a part to be tested is conveyed on the conveyor belt, the target detection method can be directly determined based on the category of the part. Alternatively, the color of the part can be determined in real time from the target image, thereby determining the target detection method for the part to be tested.

[0041] In one implementation, when the target detection method is the first detection method, the determination of whether the part to be tested is a defective part is based on the following steps: Figure 2 As shown, steps S21-S23 are included:

[0042] S21: Split the target image into LAB images.

[0043] S22: Obtain the corresponding gray area in the LAB image by using the gray threshold of the part to be tested.

[0044] S23: Determine whether the part to be tested is a defective part based on the gray area and the preset area.

[0045] Specifically, in steps S21-S23, the RGB color space of the target image is first converted to the LAB color space to distinguish the colors of the parts to be tested, as shown in the following figure. Figure 3 As shown, different channels of color are used to obtain color areas and determine whether the part to be tested exists.

[0046] The steps to convert the RGB color space of the target image to the LAB color space are as follows: Normalize the RGB color values ​​to the range [0,1], i.e., R′=R / 255, G′=G / 255, B′=B / 255, where R, G, and B represent the initial R, G, and B values ​​of the target image, and R′, G′, and B′ represent the normalized R, G, and B values; Center the RGB values: R"=(R′-5)*255 / 256, G"=(G′-5)*255 / 256, B"=(B′-5)*255 / 256, where R", G", and B" represent the centered R, G, and B values, respectively; Finally, calculate the value of LAB: L = (1 / 255)*(R"+B"+G"), a = (1 / 510)*(R"-G"+2*B"), b = (1 / 510)*(-R"+2*G"-B"). Further, a region of interest (ROI) is drawn, and the corresponding grayscale area is obtained within the ROI using the grayscale threshold of the part to be tested. If the difference between the grayscale area and the preset area is greater than the preset difference, the part to be tested is determined to be missing and not present in the target image, thus belonging to a defective part. If the difference between the grayscale area and the preset area is less than or equal to the preset difference, the part to be tested is determined to be located in the target image and is not a defective part. The preset area can be the standard area of ​​the part to be tested.

[0047] In this embodiment of the application, the absence of the part under test is determined by comparing the standard grayscale area of ​​the part under test with the actual grayscale area of ​​the part under test in the target image. This method can more accurately determine whether the part under test is missing, thereby improving detection efficiency and accuracy.

[0048] In another implementation, when the target detection method is the second detection method, the part missing detection model is the first deep learning model. Specifically, sample data of parts belonging to category 2 are acquired, including normal sample data and abnormal sample data. Normal sample data consists of images with parts, and abnormal sample data consists of images without parts. The parts are segmented and labeled, and deep learning training is performed to generate the first deep learning model.

[0049] Specifically, if the first deep learning model does not detect the part to be tested, it determines that the part to be tested is missing; if the first deep learning model detects the part to be tested, it determines that the part to be tested is located in the region of interest and obtains the outer contour of the part to be tested.

[0050] In this embodiment, since the color of the part to be tested is similar to the background color, detection by grayscale is prone to misjudgment. Therefore, a first deep learning model is used to annotate the target image to detect whether the target image includes the part to be tested, thereby improving detection efficiency and accuracy.

[0051] II. The part inspection model is a part offset detection model.

[0052] In some embodiments, the target detection method of the part offset detection model is determined by determining whether there are fixed structural members around the part to be tested.

[0053] Specifically, if there are fixed structural components around the part to be tested, the target detection method of the part offset detection model is determined to be the third detection method; if there are no fixed structural components around the part to be tested, the target detection method of the part offset detection model is determined to be the fourth detection method.

[0054] In this embodiment, the presence of fixed structural components around the part can be determined in advance based on the part's type. When the part to be tested is conveyed on the conveyor belt, the target detection method for the part can be directly determined based on its type. Alternatively, the presence of fixed structural components around the part can be determined in real time using a target image, thereby determining the target detection method for the part.

[0055] In one implementation, when the target detection method is the third detection method, the determination of whether the part to be tested is a defective part is made by the following method: Figure 4 As shown, it includes the following steps:

[0056] S31. Obtain the outer contour and minimum bounding rectangle of the part to be measured.

[0057] The outer contour of the part under test can be obtained through the first deep learning model; the minimum bounding rectangle of the part under test is determined based on a pre-stored calibration table. The pre-stored calibration table includes the mechanical point coordinates of the camera movement, image coordinates, and real-world coordinates. The real-world coordinates of the part under test are determined by the pre-stored calibration table, and thus the minimum bounding rectangle of the part under test can be determined by the real-world coordinates of the part under test.

[0058] S32. Determine whether the part under test is offset by the characteristic relationship between the outer contour and / or the smallest bounding rectangle of the part under test and the fixed structural member or fixed straight line.

[0059] The characteristic relationship includes at least one of the following:

[0060] The distance between the outer contour of the part to be measured and a fixed straight line;

[0061] The angle difference between the smallest bounding rectangle of the part to be measured and the fixed straight line;

[0062] The distance between the outer contour of the part to be measured and the fixed structural component;

[0063] The angle difference between the minimum bounding rectangle of the part under test and the minimum bounding rectangle of the fixed structure;

[0064] The obstruction relationship between the part to be tested and the fixed structural component;

[0065] The intersection relationship between the part to be tested and the fixed structural component.

[0066] For example, if the shortest distance (or longest distance) between the outer contour of a standard part and a fixed straight line is A, and the shortest distance (or longest distance) between the outer contour of the part to be measured and the fixed straight line is B, and the difference between A and B is greater than a preset difference, then it can be determined that the part to be measured has shifted, causing a change in the distance between the standard part and the fixed straight line. As another example, if the angle difference between the minimum bounding rectangle of the part to be measured and the fixed straight line is 20 degrees, and the angle difference between the minimum bounding rectangle of the standard part and the fixed straight line is 10 degrees, the deviation is too significant, and it can be determined that the part to be measured has shifted. Furthermore, if there is no obstruction or intersection relationship between the standard part and the fixed structural frame, but the part to be measured obstructs or intersects the fixed structural component, then it indicates that the part to be measured has shifted.

[0067] The above-described feature relationships are merely illustrative examples and are not limited to the examples in the specification. Other feature relationships that are of the same principle as the embodiments of this application are included within the protection scope of the embodiments of this application. Furthermore, two or more feature relationships can be used in combination to improve the accuracy of offset determination.

[0068] S33. Determine whether the part under test is a defective part based on whether the part under test has shifted.

[0069] Specifically, if the part under test shifts, it is determined to be a defective part; if the part under test does not shift, it is determined to be a qualified part.

[0070] In the embodiments of this application, when there are fixed structural members around the part to be tested, since the position of the fixed structural members is fixed, it is possible to determine whether the part to be tested has shifted by judging the distance, angle or other characteristic relationship between the part to be tested and the fixed structure or between the fixed straight line of the fixed structure.

[0071] In another implementation, when the target detection method is the fourth detection method, the determination of whether the part to be tested is a defective part is made by the following method: Figure 5 As shown, it includes the following steps:

[0072] S41. Based on the image coordinates of the part to be tested and the mechanical coordinates of the camera, calculate the real-world coordinates of the part to be tested using a pre-stored coordinate calibration table.

[0073] Specifically, the real-world coordinates of the part under test are calculated using the bilinear interpolation formula. For example... Figure 6 As shown, the default grid width is 1, Q 11 The coordinates of Q are (0,0). 11 Q 12 Q 21 Q 22 Given the coordinates of point P, where P is (x, y), then the real-world coordinates of point P are: P = (1-x)(1-y)Q 11 +(1-x)yQ 21 +x(1-y)Q 12 +xyQ 22 .

[0074] Furthermore, the top, bottom, left, and right outer contour boundaries of the part under test can be determined to their real-world coordinates through transformation.

[0075] S42. Based on the comparison between the real world coordinates of the part under test and the standard world coordinates of the part under test, determine whether the position of the part under test has shifted.

[0076] S43. Determine whether the part under test is a defective part based on whether the position of the part under test has shifted.

[0077] Specifically, if the real-world coordinates of the part under test deviate from the standard world coordinates of the part under test by more than a certain amount (this amount is given by the technician), it is determined that the position of the part under test is abnormal and that an offset has occurred.

[0078] In some embodiments, when the position of the part under test is offset, the offset dimension can also be calculated so that the technician can adjust the position of the part under test.

[0079] In the above implementation, the pre-stored calibration table is generated in the following way: the mechanical points of the camera movement are obtained; based on the distance of the camera movement, the image coordinates of the fixed checkerboard corner points on the current image are determined, and the real-world coordinates of the checkerboard corner points are obtained; a calibration table is established based on the coordinates of the mechanical points of the camera movement, the image coordinates of the current points, and the real-world coordinates.

[0080] Specifically, the calibration plate is positioned and kept stationary. Images are acquired based on the points where the camera will move along the calibration plate, as set in the configuration. Based on the acquired images of the calibration plate, the coordinates of the camera's movement are calculated, and this coordinate is assigned to the mechanical point P. M (x, y); Based on the distance the machine moves, determine the approximate range of the checkerboard corner point on the current image where the camera moves a fixed step, and obtain the image coordinates P of that corner point. p (x, y), and simultaneously record the real-world coordinates P of the corner points of the chessboard. W (x, y), where the origin coordinates are taken as the top left position of the chessboard grid; a calibration table is established, which includes the mechanical point coordinates P of the camera movement. M (x, y), the image coordinates P of the current point. p (x,y), and real-world coordinates P W (x,y), as shown below:

[0081]

[0082] In this embodiment of the application, by establishing a calibration table, the real-world coordinates of the part to be tested can be determined in real time based on the calibration table, thereby determining the part features such as the outer contour and center of gravity of the part to be tested.

[0083] III. The part inspection model is a part defect detection model.

[0084] When the part detection model is a part defect detection model, and the part defect detection model is a second deep learning model, the target image is detected by the second deep learning model, and an annotated image is output. Based on whether there are defects marked in the annotated image, it is determined whether the part under test is a defective part.

[0085] The second deep learning model is trained on defective and normal samples. When detecting defects in parts, the second deep learning model is used to infer whether there are defects such as wrinkles or breaks in the target image. If no defects are found, the part to be tested is determined to be a qualified part. If wrinkles or breaks are detected, the abnormal features such as length and width of the part are obtained, and the part to be tested is determined to be a defective part.

[0086] In this embodiment of the application, by using a deep learning model to detect whether there are defects in the part under test, the part under test can be comprehensively detected, thereby improving the detection accuracy and speed.

[0087] In the three embodiments described above, those skilled in the art can combine any two or three embodiments according to actual needs. For example, a missing part detection model can be used to detect whether the part under test is missing. If the part under test is not missing, a part offset detection model can be used to further detect whether the part under test is offset. If the part under test is not offset, it is determined to be a qualified part; if it is offset, it is determined to be a defective part. As another example, a part offset detection model can be used to detect whether the part under test is offset. If it is not offset, a part defect detection model can be used to further detect the part under test. If a defect exists in the part under test, it is determined to be a defective part; if no defect exists, it is determined to be a qualified part.

[0088] The following describes the use of the three embodiments in combination. For example... Figure 7 As shown in the embodiments of this application, a vision-based method for detecting defective parts is also provided, including the following steps:

[0089] S51. Detect whether the part to be tested is missing based on the part missing detection model.

[0090] S521. If the missing part detection model detects that the part to be tested is not missing, the part offset model is used to detect whether the part to be tested is offset.

[0091] S522. If the missing part detection model detects that the part to be tested is missing, it confirms that the part to be tested is a defective part.

[0092] S531. If the part offset detection model detects that the part under test has not been offset, the part defect detection model is used to detect whether there is a defect in the part under test.

[0093] S532. If the part offset detection model detects that the part under test has offset, it determines that the part under test is a defective part.

[0094] S541. If the part defect detection model detects a defect in the part to be tested, the part to be tested is determined to be a defective part.

[0095] S542. If the part defect detection model detects that there are no defects in the part to be tested, the part to be tested is determined to be a qualified part.

[0096] The specific implementation methods of each step in steps S51-S542 can be referred to the specific implementation methods in the above embodiments, and will not be repeated here in the embodiments of this application.

[0097] Furthermore, the part offset detection model and the part defect detection model can be swapped. If the part missing detection model detects that the part under test is not missing, the part defect detection model is first used to detect whether the part under test has a defect. If it is determined that the part under test does not have a defect, the part offset detection model is used to detect whether the part under test has been offset. If it has been offset, the part under test is confirmed to be a defective part. If it has not been offset, the part under test is determined to be a qualified part.

[0098] In the embodiments of this application, the parts under test are detected by the missing part detection model, the offset part detection model, and the defect part detection model, respectively. This can determine whether the parts under test are missing, offset, or defective, thereby enabling comprehensive detection of problems in the parts under test and improving detection efficiency and accuracy.

[0099] In order to Figure 7 The embodiments shown will be described in more detail below. Figure 8 Explanation: When a part to be tested is conveyed on the conveyor belt, the target detection method determines whether a part is missing by distinguishing its color. If the saturation difference between the color of the part to be tested and the background color is significant, LAB image processing is used to determine whether the part is missing; if the saturation difference is not significant, a deep learning model is used. If the part to be tested is determined to be missing, it is identified as a defective part. If the part to be tested is determined to be not missing, the target detection method is determined by whether there are fixed structural components around it. If there are fixed structural components around the part to be tested, the feature relationship between the part to be tested and the structural components is used to determine whether the part has shifted; if there are no fixed structural components around the part to be tested, the real-world coordinates of the part to be tested are obtained through bilinear interpolation, and the relationship between the real-world coordinates of the part to be tested and the real-world coordinates of the standard part is used to determine whether the part has shifted. If the part to be tested is determined to have shifted, it is identified as a defective part, and detection stops; if the part to be tested is determined not to have shifted, a deep learning detection model is used to detect the part to determine whether it has a defect. If the part under test has a defect, it is determined to be a defective part and the test is stopped; if it is determined that the part under test has no defect, it is determined to be a qualified part and the test result is output.

[0100] In this embodiment, the true information of the parts can be accurately calculated, ensuring that qualified precision parts are provided for the product.

[0101] In some embodiments of this application, a vision-based defective parts detection device is provided, which corresponds one-to-one with the vision-based defective parts detection methods described in the above embodiments. For example... Figure 9 As shown, the vision-based defective parts detection device includes a data acquisition module 101 and a detection module 102.

[0102] Acquisition module 101 is used to acquire target images of regions of interest, including the standard area where the part to be tested is located;

[0103] The detection module 102 is used to detect the target image based on the part detection model to determine whether the part to be tested is a defective part;

[0104] The part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part under test is missing based on the color of the part under test, the part offset detection model is used to detect whether the part under test is offset based on the part features of the part under test, and the part defect detection model is used to detect whether the part under test has a defect based on a deep learning model.

[0105] In some embodiments of this application, when the part detection model in the above-described apparatus is a part missing detection model, the detection module 102 is specifically used to determine the target detection method of the part missing detection model based on the color of the part to be tested; and to determine whether the part to be tested is a defective part based on the target detection method of the part missing detection model.

[0106] In some embodiments of this application, in the above-described apparatus, the detection module 102 is specifically used to determine the target detection method of the missing part detection model as a first detection method if the saturation of the color of the part to be tested and the background color of the target image is greater than a threshold; and to determine the target detection method of the missing part detection model as a second detection method if the saturation of the color of the part to be tested and the background color of the target image is less than or equal to a threshold.

[0107] In some embodiments of this application, when the target detection method is the first detection method in the above-described apparatus, the detection module 102 is specifically used to split the target image into LAB images; obtain the corresponding gray area in the LAB image through the gray area threshold of the part to be tested; and determine whether the part to be tested is a defective part based on the gray area and the preset area.

[0108] In some embodiments of this application, in the above-described device, the detection module 102 is specifically used to determine that the part to be tested is not a defective part if the difference between the gray area and the preset area is less than or equal to a preset difference; and to determine that the part to be tested is a defective part if the difference between the gray area and the preset area is greater than a preset difference.

[0109] In some embodiments of this application, when the target detection method in the above-described apparatus is the second detection method, the missing part detection model is the first deep learning model. The detection module 102 is specifically used to detect the target image based on the first deep learning model. If the first deep learning model does not detect the part to be tested, it determines that the part to be tested is missing; or to detect the target image based on the first deep learning model. If the first deep learning model detects the part to be tested, it determines that the part to be tested is located in the region of interest and obtains the outer contour of the part to be tested.

[0110] In some embodiments of this application, when the part detection model is a part offset detection model in the above-described apparatus, the detection module 102 is specifically used to determine the target detection method of the part offset detection model based on whether there are fixed structural components around the part to be tested; and to detect the target image based on the target detection method of the part offset detection model to determine whether the part to be tested is a defective part.

[0111] In some embodiments of this application, in the above-described apparatus, the detection module 102 is specifically used to determine the target detection method of the part offset detection model as the third detection method if there are fixed structural components around the part to be tested; or to determine the target detection method of the part offset detection model as the fourth detection method if there are no fixed structural components around the part to be tested.

[0112] In some embodiments of this application, in the above-described apparatus, the target detection method is a third detection method. The detection module 102 is specifically used to obtain the outer contour and minimum bounding rectangle of the part to be tested; based on the characteristic relationship between the outer contour and / or minimum bounding rectangle of the part to be tested and the fixed structural member or fixed straight line, determine whether the part to be tested is offset; and based on whether the part to be tested is offset, determine whether the part to be tested is a defective part.

[0113] In some embodiments of this application, in the above-described apparatus, the characteristic relationships include at least one of the following: the distance between the outer contour of the part to be measured and the fixed straight line; the angle difference between the minimum bounding rectangle of the part to be measured and the fixed straight line; the distance between the outer contour of the part to be measured and the fixed structural member; the angle difference between the minimum bounding rectangle of the part to be measured and the minimum bounding rectangle of the fixed structural member; the occlusion relationship between the part to be measured and the fixed structural member; and the intersection relationship between the part to be measured and the fixed structural member.

[0114] In some embodiments of this application, in the above-described apparatus, the minimum circumscribed rectangle of the part under test is determined based on a pre-stored calibration table.

[0115] In some embodiments of this application, in the above-described apparatus, the target detection method is a fourth detection method. The detection module 102 is specifically used to calculate the real-world coordinates of the part under test based on the image coordinates of the part under test and the mechanical coordinates of the camera, through a pre-stored coordinate calibration table; compare the real-world coordinates of the part under test with the standard world coordinates of the part under test to determine whether the position of the part under test has shifted; and determine whether the part under test is a defective part based on whether the position of the part under test has shifted.

[0116] In some embodiments of this application, in the above-described apparatus, the standard world coordinates of the part under test are determined based on a pre-stored calibration table.

[0117] In some embodiments of this application, in the above-described apparatus, the pre-stored calibration table is generated based on the following method: obtaining the mechanical points of camera movement; determining the image coordinates of the fixed checkerboard corner points on the current image based on the distance of camera movement, and obtaining the real-world coordinates of the checkerboard corner points; establishing a calibration table based on the coordinates of the mechanical points of camera movement, the image coordinates of the current points, and the real-world coordinates.

[0118] In some embodiments of this application, when the part detection model is a part defect detection model in the above-described apparatus, the part defect detection model is a second deep learning model. The detection module 102 is specifically used to detect the target image based on the second deep learning model and output an annotated image; based on whether there is a defect marked in the annotated image, it is determined whether the part to be tested is a defective part.

[0119] In some embodiments of this application, in the above-described apparatus, the part detection model includes a part missing detection model, a part offset detection model, and a part defect detection model. The detection module 102 is specifically used to detect whether the part to be tested is missing based on the part missing model; if the part missing detection model detects that the part to be tested is not missing, it detects whether the part to be tested is offset based on the part offset model; if the part offset model detects that the part to be tested is not offset, it detects whether the part to be tested is a defective part based on the part defect detection model.

[0120] In some embodiments of this application, in the above-described apparatus, the detection module 102 is specifically used to determine that the part under test is a defective part if the part offset model detects that the part under test is not offset and the part defect detection model detects that the part under test has a defect; or if the part offset model detects that the part under test is not offset and the part defect detection model detects that the part under test does not have a defect, the part under test is determined not to be a defective part. The detection of whether the part under test is a defective part is based on the part defect detection model.

[0121] It should be noted that any of the aforementioned vision-based defective parts detection devices can implement the aforementioned vision-based defective parts detection methods one by one, which will not be elaborated here.

[0122] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Figure 10 As shown, at the hardware level, this electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may include non-volatile memory, such as at least one disk drive. Of course, this electronic device may also include other hardware required for other business operations.

[0123] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0124] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0125] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a vision-based defective part detection device at the logical level. The processor executes the program stored in memory and specifically performs the aforementioned method.

[0126] The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0127] This electronic device can execute the vision-based defective part detection method provided in several embodiments of this application, and is implemented as a vision-based defective part detection device. Figure 9 The functions of the embodiments shown are not described in detail here.

[0128] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform the vision-based defective part detection method provided in several embodiments of this application.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A vision-based method for detecting defective parts, characterized in that, The method includes: Acquire a target image of the region of interest, which includes the standard area where the part to be tested is located; The target image is detected based on a part detection model to determine whether the part to be tested is a defective part; The part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part under test is missing based on the color of the part under test; the part offset detection model is used to detect whether the part under test is offset based on the part features of the part under test; and the part defect detection model is used to detect whether the part under test has a defect based on a deep learning model. When the part detection model is a part missing detection model, the step of detecting the target image based on the part detection model to determine whether the part to be tested is a defective part includes: Based on the color of the part to be tested, determine the target detection method of the part missing detection model; Based on the target detection method of the missing part detection model, it is determined whether the part to be tested is a defective part. The step of determining the target detection method of the missing part detection model based on the color of the part to be tested includes: If the difference in saturation between the color of the part to be tested and the background color of the target image is greater than a threshold, the target detection method of the part missing detection model is determined to be the first detection method. If the difference in saturation between the color of the part to be tested and the background color of the target image is less than or equal to a threshold, the target detection method of the part missing detection model is determined to be the second detection method. When the target detection method is the first detection method, the target detection method based on the part missing detection model determines whether the part to be tested is a defective part, including: Convert the target image into a LAB image; The gray area in the LAB image is obtained by using the gray threshold of the part under test; Based on the gray area and the preset area, determine whether the part to be tested is a defective part; When the target detection method is the second detection method, the part missing detection model is the first deep learning model; the target detection method based on the part detection model, determining whether the part to be tested is a defective part, includes: The target image is detected based on a first deep learning model. If the first deep learning model does not detect the part to be tested, it is determined that the part to be tested is missing; or The target image is detected based on the first deep learning model. If the first deep learning model detects the part to be tested, it is determined that the part to be tested is located in the region of interest, and the outer contour of the part to be tested is obtained.

2. The method according to claim 1, characterized in that, The step of determining whether the part to be tested is a defective part based on the gray area and the preset area includes: If the difference between the gray area and the preset area is less than or equal to the preset difference, the part to be tested is determined not to be a defective part. If the difference between the gray area and the preset area is greater than the preset difference, the part to be tested is determined to be a defective part.

3. The method according to claim 1, characterized in that, When the part detection model is a part offset detection model, the step of detecting the target image based on the part detection model to determine whether the part to be tested is a defective part includes: The target detection method of the part offset detection model is determined based on whether there are fixed structural components around the part to be tested. The target image is detected using the target detection method based on the part offset detection model to determine whether the part to be tested is a defective part.

4. The method according to claim 3, characterized in that, The method for determining the target detection method of the part offset detection model based on whether there are fixed structural components around the part to be tested includes: If there are fixed structural components around the part to be tested, the target detection method of the part offset detection model is determined to be the third detection method; or If there are no fixed structural components around the part to be tested, the target detection method of the part offset detection model is determined to be the fourth detection method.

5. The method according to claim 4, characterized in that, The target detection method is a third detection method. This target detection method, based on the part offset detection model, detects the target image to determine whether the part to be tested is a defective part, including: Obtain the outer contour and minimum bounding rectangle of the part to be measured; Based on the characteristic relationship between the outer contour and / or minimum bounding rectangle of the part to be tested and the fixed structural member or fixed straight line, determine whether the part to be tested is offset; Whether the part under test is defective is determined based on whether the part under test has shifted.

6. The method according to claim 5, characterized in that, The characteristic relationship includes at least one of the following: The distance between the outer contour of the part to be measured and a fixed straight line; The angle difference between the smallest bounding rectangle of the part to be measured and the fixed straight line; The distance between the outer contour of the part to be measured and the fixed structural component; The angle difference between the minimum bounding rectangle of the part under test and the minimum bounding rectangle of the fixed structure; The obstruction relationship between the part to be tested and the fixed structural component; The intersection relationship between the part to be tested and the fixed structural component.

7. The method according to claim 6, characterized in that, The minimum bounding rectangle of the part to be tested is determined based on a pre-stored coordinate calibration table.

8. The method according to claim 7, characterized in that, The target detection method is the fourth detection method. This target detection method, based on the part detection model, detects the target image to determine whether the part to be tested is a defective part, including: Based on the image coordinates of the part under test and the mechanical coordinates of the camera, the real-world coordinates of the part under test are calculated using a pre-stored coordinate calibration table. By comparing the real-world coordinates of the part under test with the standard world coordinates of the part under test, it is determined whether the position of the part under test has shifted. Whether the part under test is defective is determined based on whether its position has shifted.

9. The method according to claim 8, characterized in that, The standard world coordinates of the part to be tested are determined based on a pre-stored coordinate calibration table.

10. The method according to any one of claims 7-9, characterized in that, The pre-stored coordinate calibration table is generated based on the following method: Obtain the mechanical positions of the moving camera; Based on the distance the camera moves, determine the image coordinates of the checkerboard corner points on the current image at a fixed step size, and obtain the real-world coordinates of the checkerboard corner points; A coordinate calibration table is established based on the mechanical position coordinates of the camera movement, the image coordinates of the current position, and the real-world coordinates.

11. The method according to claim 1, characterized in that, When the part detection model is a part defect detection model, the part defect detection model is a second deep learning model; The step of detecting the target image based on the part detection model to determine whether the part to be tested is a defective part includes: The target image is detected based on the second deep learning model, and an annotated image is output. Based on whether defects are marked in the labeled image, it is determined whether the part to be tested is a defective part.

12. The method according to claim 1, characterized in that, The part detection model includes a part missing detection model, a part offset detection model, and a part defect detection model. The step of detecting the target image based on the part detection model to determine whether the part to be tested is a defective part includes: The missing part detection model is used to detect whether the part to be tested is missing. If the missing part detection model detects that the part to be tested is not missing, the part offset detection model is used to detect whether the part to be tested is offset. If the part offset detection model detects that the part under test has not been offset, the part defect detection model is used to determine whether the part under test is a defective part.

13. The method according to claim 12, characterized in that, If the part offset detection model detects that the part under test has not offset, the step of detecting whether the part under test is a defective part based on the part defect detection model includes: If the part offset detection model detects that the part under test is not offset, and the part defect detection model detects that the part under test has a defect, then the part under test is determined to be a defective part; or If the part offset detection model detects that the part under test is not offset, and the part defect detection model detects that the part under test has no defects, then the part under test is determined not to be a defective part. The part defect detection model is used to determine whether the part under test is a defective part.

14. A vision-based defective parts detection device, characterized in that, The apparatus for performing the vision-based defective part detection method according to any one of claims 1 to 13, the apparatus comprising: The acquisition module is used to acquire target images of the region of interest, which includes the standard area where the part to be tested is located. The detection module is used to detect the target image based on the part detection model to determine whether the part to be tested is a defective part; The part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part under test is missing based on the color of the part under test, the part offset detection model is used to detect whether the part under test is offset based on the part features of the part under test, and the part defect detection model is used to detect whether the part under test has a defect based on a deep learning model.

15. An electronic device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the vision-based defective parts detection method as described in any one of claims 1-13.

16. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the steps of the vision-based defective parts detection method as described in any one of claims 1-13.

Images