Defective part detection method and device based on vision

Abstract

The invention relates to the technical field of image processing, and discloses a vision-based bad part detection method and device. The method comprises the steps of collecting a target image of a region of interest, wherein the region of interest comprises a standard region where a to-be-detected part is located; detecting the target image based on a part detection model, and determining whether the to-be-detected part belongs to a bad part or not; wherein the part detection model comprises at least one of a part missing detection model, a part offset detection model and a part defect detection model; the part missing detection model detects whether a to-be-detected part is missing or not based on the color of the to-be-detected part, and the part offset detection model is used for detecting whether the to-be-detected part is offset or not based on the part characteristics of the to-be-detected part. The part defect detection model is used for detecting whether the part to be detected has defects or not based on a deep learning model. The detection efficiency, the detection precision and the detection time of defective parts can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a vision-based defective parts detection method and device. Background Art

[0002] With the continuous development of the manufacturing industry, ensuring product quality has become a key competitive advantage. Accurately positioning components on the product's mid-panel is crucial for quality control. Any misalignment can affect the assembly of other components in subsequent processes. Therefore, incorporating quality inspection procedures, whether manual or industrial visual inspection, into production processes is crucial to ensuring product quality and reducing rework between different steps.

[0003] Currently, the installation of parts on mid-panels in factories generally relies on manual visual inspection or measurement to determine whether each material is misaligned or whether the misalignment is within a reasonable range. This method has a series of problems, including limited measurement accuracy, inconsistent standards among different personnel, and strong subjectivity leading to missed inspections or misjudgments. Summary of the Invention

[0004] In response to the above situation, the embodiments of the present application provide a vision-based defective parts detection method and device, which aims to solve the above problems or at least partially solve the above problems.

[0005] In the first aspect, an embodiment of the present application provides a vision-based defective parts detection method, characterized in that the method includes: acquiring a target image of an area of ​​interest, wherein the area of ​​interest includes a standard area where the part to be tested is located; detecting the target image based on a part detection model to determine whether the part to be tested is a defective part; wherein the part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested is defective based on a deep learning model.

[0006] In the second aspect, an embodiment of the present application also provides a vision-based defective parts detection device, the device comprising: an acquisition module for acquiring a target image of an area of ​​interest, the area of ​​interest including a standard area where the part to be tested is located; a detection module for detecting the target image based on a part detection model to determine whether the part to be tested is a defective part; wherein the part detection model comprises at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested is defective based on a deep learning model.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to perform the steps of the first aspect described above.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device performs the steps of the first aspect above.

[0009] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0010] By capturing a target image of the standard area where the part to be tested is located and processing and testing it based on the part detection model, it is determined whether the part to be tested is a defective part. Since the part detection model includes a part missing detection model, a part offset detection model, and a part missing detection model, it can detect whether the part to be tested is missing, whether the part to be tested is offset, and whether the part to be tested has defects. This allows for accurate detection of the true condition of the part to be tested, ensuring the provision of precision parts of appropriate quality for the product. At the same time, using the model to test parts saves manpower, material resources, and time, and improves detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A schematic diagram of the process of a vision-based defective parts detection method provided in an embodiment of the present application is shown;

[0013] Figure 2 A flowchart of a vision-based defective parts detection method provided by another embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram showing the effect of converting RGB to LAB channels provided by an embodiment of the present application is shown;

[0015] Figure 4 A flowchart of a vision-based defective parts detection method provided by another embodiment of the present application is shown;

[0016] Figure 5 A flowchart of a vision-based defective parts detection method provided by another embodiment of the present application is shown;

[0017] Figure 6 A schematic diagram of the bilinear difference principle provided in an embodiment of the present application is shown.

[0018] Figure 7 A flowchart of a vision-based defective parts detection method provided by another embodiment of the present application is shown;

[0019] Figure 8 A flowchart of a vision-based defective parts detection method provided by another embodiment of the present application is shown;

[0020] Figure 9 The structure diagram of the defective parts detection device based on vision provided by an embodiment of the present application is shown;

[0021] Figure 10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that such usage is interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to."

[0024] As described in the background technology, the installation of parts on the middle plate in the factory currently generally needs to rely on manual visual inspection or manual measurement to judge whether there is any deviation for each material or whether the deviation is within a reasonable range. This method has a series of problems, including limitations on measurement accuracy, inconsistent standards of different personnel, and strong subjectivity leading to missed detection or misjudgment.

[0025] Based on this, in order to more stably and efficiently solve the tedious operations of solving the defects of existing mid-plate parts and improve the defects of the original mid-plate parts defect detection, the present application provides a vision-based defective parts detection method, which collects the target image of the standard area where the part to be tested is located, and processes and detects the target image based on the part detection model to determine whether the part to be tested is a defective part.

[0026] The present application is described in detail below through specific embodiments.

[0027] Figure 1 The flowchart of a defective parts detection method based on vision provided by the embodiment of the present application is shown. Figure 1 It can be seen that this application at least includes steps S101 and S102:

[0028] Step S11: Acquire a target image of the region of interest.

[0029] The region of interest includes the standard area where the part to be tested is located. The standard area refers to the area where the standard part is located. To avoid errors, the region of interest is an appropriately expanded area of ​​the standard area. The expanded area can be specified by the technician.

[0030] Step S12: Detect the target image based on the part detection model to determine whether the part to be detected is a defective part.

[0031] Among them, the part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model.

[0032] Specifically, the missing parts detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested is defective based on the deep learning model.

[0033] It is worth noting that the model involved in this application does not necessarily represent a machine learning model, but is only a high-level concept of a processing process. The specific process of model detection can be a machine learning process or an implementation process of a processing method.

[0034] from Figure 1As can be seen from the method shown, the present application determines whether the part to be tested is a defective part by collecting a target image of the standard area where the part to be tested is located and processing and detecting the target image based on the part detection model. Since the part detection model includes a part missing detection model, a part offset detection model, and a part missing detection model, it can respectively detect whether the part to be tested is missing, whether the part to be tested is offset, and whether the part to be tested has defects, thereby accurately detecting the true condition of the part to be tested, ensuring that precision parts of appropriate quality are provided for the product, and at the same time, detecting parts through the model saves manpower, material resources, and time, and improves detection accuracy.

[0035] The following describes how to determine whether the part to be tested is a defective part in step 12 when the part detection models are a part missing detection model, a part offset detection model, and a part defect detection model.

[0036] 1. The parts detection model is a parts missing detection model

[0037] In some embodiments, the target detection mode of the part missing detection model is determined by the color of the part to be tested, and whether the part to be tested is a defective part is detected based on the target detection mode.

[0038] Specifically, if the saturation of the color of the part to be tested and the background color of the target image is greater than a threshold, the part category of the part to be tested is determined to be category 1, and the target detection method of the missing parts detection model is determined to be the first detection method. If the saturation of the color of the part to be tested and the background color of the target image is less than or equal to the threshold, the part category of the part to be tested is determined to be category 2, and the target detection method of the missing parts detection model is determined to be the second detection method.

[0039] For example, if the background color is white and the color of the part to be tested is a distinct color such as yellow or red, such as a bright yellow copper sheet or a red rubber sleeve, then the part category of the part to be tested can be determined to be category 1, and the corresponding target detection method is the first detection method; if the color of the part to be tested is white, beige, or other colors that are not clearly distinguishable from the background color, then the part category of the part to be tested can be determined to be category 2, and the corresponding target detection method is the second detection method.

[0040] In the embodiments of the present application, each part can be pre-classified based on its color. When the part is conveyed on the conveyor belt, the target detection method for the part can be determined directly based on its category. Alternatively, the color of the part can be determined in real time from the target image, thereby determining the target detection method for the part.

[0041] In one embodiment, when the target detection mode is the first detection mode, it is determined whether the part to be detected is a defective part based on the following steps: Figure 2 As shown, steps S21-S23 are included:

[0042] S21: Split the target image into LAB images.

[0043] S22: Obtain the corresponding grayscale area in the LAB image through the grayscale threshold of the part to be tested.

[0044] S23: Determine whether the part to be tested is a defective part based on the grayscale area and the preset area.

[0045] Specifically, in steps S21-S23, the RGB color space of the target image is first converted to the LAB color space to distinguish the color of the part to be tested. The result is as follows: Figure 3 As shown, the colors of different channels are used to obtain the color area to determine whether the part to be tested exists.

[0046] The steps to convert the RGB color space of the target image to the LAB color space are as follows: normalize the RGB color values ​​to the range of [0,1], that is, R'=R / 255, G'=G / 255, B'=B / 255, where R, G, B represent the initial R, G, B values ​​of the target image, and R', G', B' represent the normalized R, G, B values; center the RGB values: R"=(R'-5)*255 / 256, G"=(G'-5)*255 / 256, B"=(B'-5)*255 / 256, where R", G", B" represent the centered R, G, B values ​​respectively; then calculate the value of LAB: L=(1 / 255)*(R"+B"+G"), a=(1 / 510)*(R"-G"+2*B"), b=(1 / 510)*(-R"+2*G"-B"). Further, a region of interest is drawn, and the corresponding grayscale area is obtained in the region of interest by using the grayscale threshold of the part to be tested. If the difference between the grayscale area and the preset area is greater than the preset difference, it is determined that the part to be tested is missing and does not exist in the target image, and is a defective part; if the difference between the grayscale area and the preset area is less than or equal to the preset difference, it is determined that the part to be tested is located in the target image and is not a defective part. The preset area can be the standard area of ​​the part to be tested.

[0047] In an embodiment of the present application, by judging the standard grayscale area of ​​the part to be tested and the actual grayscale area of ​​the part to be tested in the target image to determine whether the part to be tested is missing, it is possible to more accurately determine whether the part to be tested is missing, thereby improving detection efficiency and accuracy.

[0048] In another embodiment, when the target detection mode is the second detection mode, the missing parts detection model is a first deep learning model. Specifically, sample data of part category 2 is obtained, including normal sample data and abnormal sample data. Normal sample data is images with parts, and abnormal sample data is images without parts. The parts are segmented and labeled, and deep learning training is performed to generate the first deep learning model.

[0049] Specifically, if the first deep learning model does not detect the part to be measured, it is determined that the part to be measured is missing; if the first deep learning model detects the part to be measured, it is determined that the part to be measured is located in the region of interest, and the outer contour of the part to be measured is obtained.

[0050] In an embodiment of the present application, since the color of the part to be tested is similar to the background color, misjudgment is prone to occur when detecting through grayscale. Therefore, the target image is annotated by a first deep learning model to detect whether the target image includes the part to be tested, thereby improving detection efficiency and accuracy.

[0051] 2. The part detection model is a part offset detection model

[0052] In some embodiments, the target detection mode of the part offset detection model is determined by determining whether there are fixed structural parts around the part to be detected.

[0053] Specifically, if there are fixed structural parts around the part to be tested, the target detection method of the part offset detection model is determined to be the third detection method; if there are no fixed structural parts around the part to be tested, the target detection method of the part offset detection model is determined to be the fourth detection method.

[0054] In the embodiments of the present application, the presence of fixed structural components around a part can be determined in advance based on the part type. When a part is conveyed on a conveyor belt, the target detection method for the part can be determined directly based on the part type. Alternatively, the presence of fixed structural components around a part can be determined in real time based on the target image, thereby determining the target detection method for the part.

[0055] In one embodiment, when the target detection mode is the third detection mode, it is determined whether the part to be tested is a defective part by the following method, such as Figure 4 As shown, the following steps are included:

[0056] S31. Obtain the outer contour and the minimum circumscribed rectangle of the part to be measured.

[0057] The outer contour of the part to be measured can be obtained through the first deep learning model; the minimum bounding rectangle of the part to be measured is determined based on a pre-stored calibration table. The pre-stored calibration table includes the mechanical point coordinates of the camera movement, image coordinates, and real-world coordinates. The real-world coordinates of the part to be measured are determined using the pre-stored calibration table, and the minimum bounding rectangle of the part to be measured can be determined using the real-world coordinates of the part to be measured.

[0058] S32. Determine whether the part to be measured is offset based on the characteristic relationship between the outer contour and / or the minimum circumscribed rectangle of the part to be measured and the fixed structural member or the fixed straight line.

[0059] The characteristic relationship includes at least one of the following:

[0060] The distance between the outer contour of the part to be measured and the fixed straight line;

[0061] The angle difference between the minimum circumscribed rectangle of the part to be measured and the fixed straight line;

[0062] The distance between the outer contour of the part to be measured and the fixed structural part;

[0063] The angle difference between the minimum circumscribed rectangle of the part to be tested and the minimum circumscribed rectangle of the fixed structural part;

[0064] The occlusion relationship between the part to be tested and the fixed structural parts;

[0065] The intersection relationship between the part to be measured and the fixed structural part.

[0066] For example, the shortest distance (longest distance) between the outer contour of the standard part and the fixed straight line is A, and the shortest distance (longest distance) between the outer contour of the part to be measured and the fixed straight line is B. If the difference between A and B is greater than the preset difference, it can be determined that the part to be measured has shifted, causing the distance between the standard part and the fixed straight line to change. For another example, the angle difference between the minimum circumscribed rectangle of the part to be measured and the fixed straight line is 20 degrees, and the angle difference between the minimum circumscribed rectangle of the standard part and the fixed straight line is 10 degrees. It can be seen that the deviation is too obvious, and it can be determined that the part to be measured has shifted. For another example, there is no occlusion or intersection relationship between the standard part and the fixed structural frame, but the part to be measured does block or intersect the fixed structural member, which means that the part to be measured has shifted.

[0067] The above characteristic relationships are merely illustrative and are not limited to the examples in this specification. Other characteristic relationships that share the same principles as the embodiments of this application are included within the scope of protection of the embodiments of this application. In addition, two or more characteristic relationships can be used in combination to improve the accuracy of offset determination.

[0068] S33. Determine whether the part to be tested is a defective part based on whether the part to be tested is offset.

[0069] Specifically, if the part to be tested is offset, the part to be tested is determined to be a defective part; if the part to be tested is not offset, the part to be tested is determined to be a qualified part.

[0070] In an embodiment of the present application, when there are fixed structural parts around the part to be measured, since the position of the fixed structural parts is fixed, it is possible to determine whether the part to be measured has been offset by judging the distance, angle or other characteristic relationship between the part to be measured and the fixed structure or between the fixed straight lines of the fixed structural parts.

[0071] In another embodiment, when the target detection mode is the fourth detection mode, it is determined whether the part to be tested is a defective part by the following method, such as Figure 5 As shown, the following steps are included:

[0072] S41 . Calculate the real-world coordinates of the part to be measured based on the image coordinates of the part to be measured and the mechanical coordinates of the camera using a pre-stored coordinate calibration table.

[0073] Specifically, the real world coordinates of the part to be measured are calculated using a bilinear difference formula. Figure 6 As shown, the default grid width is 1, Q 11 The coordinates are (0,0), and Q 11 ,Q 12 ,Q 21 ,Q 22 The coordinates of P are (x, y), then the real world coordinates of point P are: P = (1-x)(1-y)Q 11 +(1-x)yQ 21 +x(1-y)Q 12 +xyQ 22 .

[0074] In addition, the upper, lower, left, and right outer contour boundaries of the part to be measured can be determined by transformation to determine the real-world coordinates.

[0075] S42. Determine whether the position of the part to be measured is offset based on a comparison between the real world coordinates of the part to be measured and the standard world coordinates of the part to be measured.

[0076] S43. Determine whether the part to be tested is a defective part based on whether the position of the part to be tested is offset.

[0077] Specifically, if the real world coordinates of the part to be measured and the standard world coordinates of the part to be measured deviate by more than a certain amount (this certain amount is given by a technician), it is determined that the position of the part to be measured is abnormal and deviates.

[0078] In some embodiments, when the position of the part to be measured is offset, the offset size can also be calculated so that the technician can adjust the position of the part to be measured.

[0079] In the above embodiment, the pre-stored calibration table is generated based on the following method: obtaining the mechanical point position of the camera movement; based on the distance of the camera movement, determining the image coordinates of the checkerboard corner points where the camera movement is fixed on the current image, and obtaining the real-world coordinates of the checkerboard corner points; establishing a calibration table based on the coordinates of the mechanical point position of the camera movement, the image coordinates of the current point position, and the real-world coordinates.

[0080] Specifically, place the calibration plate to ensure that it does not move, and perform image acquisition according to the point where the camera moves on the set plate; based on the captured image of the calibration plate, calculate the coordinate point where the camera moves, which is the mechanical point P M (x, y); According to the distance of mechanical movement, determine the approximate range of the checkerboard corner point of the camera moving a fixed step on the current image, and obtain the image coordinates P of the corner point coordinates p (x, y), while recording the real world coordinates P of the checkerboard corners W (x, y), where the upper left position of the chessboard is the origin coordinate; establish a calibration table, which includes the mechanical point coordinates P of the camera movement M (x,y), the image coordinates P of the current point p (x,y), and real-world coordinates P W (x,y), as shown below:

[0081]

[0082] In an embodiment of the present application, by establishing a calibration table, the real-world coordinates of the part to be measured can be determined in real time based on the calibration table, thereby determining the part features such as the outer contour and center of gravity of the part to be measured.

[0083] 3. Parts detection model is a parts defect detection model

[0084] When the part inspection model is a part defect inspection model, the part defect inspection model is a second deep learning model. The second deep learning model detects the target image, outputs a labeled image, and determines whether the part to be inspected is a defective part based on whether defects are marked in the labeled image.

[0085] The second deep learning model is trained based on defective and normal samples. When inspecting parts for defects, the second deep learning model is used to infer the presence of defects such as wrinkles and breakage on the target image. If no such defects are present, the part is deemed qualified. If wrinkles or breakage are detected, the model captures the part's abnormal features, such as length and width, and determines that the part is not defective.

[0086] In an embodiment of the present application, by utilizing a deep learning model to detect whether a part to be tested has defects, the part to be tested can be comprehensively inspected, thereby improving the accuracy and speed of detection.

[0087] In the above-mentioned embodiments 1, 2 and 3, those skilled in the art may combine any two or three of the above-mentioned embodiments according to actual needs. For example, a part missing detection model is used to detect whether the part to be tested is missing. If the part to be tested is not missing, the part offset detection model is further used to detect whether the part to be tested is offset. If the part to be tested is not offset, the part to be tested is determined to be a qualified part; if the part to be tested is offset, the part to be tested is determined to be a defective part. For another example, a part offset detection model is used to detect whether the part to be tested is offset. If the part to be tested is not offset, the part to be tested is further detected by a part defect detection model. When there is a defect in the part to be tested, the part to be tested is determined to be a defective part; if there is no defect in the part to be tested, the part to be tested is determined to be a qualified part.

[0088] The following describes the situation where the three embodiments are used in combination. Figure 7 As shown, the embodiment of the present application also provides a defective parts detection method based on vision, comprising the following steps:

[0089] S51. Detect whether the part to be tested is missing based on the part missing detection model.

[0090] S521. If the part missing detection model detects that the part to be tested is not missing, detect whether the part to be tested is offset based on the part offset model.

[0091] S522. If the part missing detection model detects that the part to be tested is missing, it is determined that the part to be tested is a defective part.

[0092] S531. If the part offset detection model detects that the part to be tested is not offset, detect whether the part to be tested has defects based on the part defect detection model.

[0093] S532: If the part offset detection model detects that the part to be tested is offset, it is determined that the part to be tested is a defective part.

[0094] S541. If the part defect detection model detects that the part to be tested has defects, it is determined that the part to be tested is a defective part.

[0095] S542. If the part defect detection model detects that the part to be tested does not have defects, it is determined that the part to be tested is a qualified part.

[0096] The specific implementation methods of each step in steps S51-S542 can refer to the specific implementation methods in the above embodiments, and the embodiments of this application will not be repeated here.

[0097] In addition, the part offset detection model and the part defect detection model can also be swapped. When the part missing detection model detects that the part to be tested is not missing, the part defect detection model is first used to detect whether the part to be tested has defects. If it is determined that the part to be tested does not have defects, the part offset detection model is further used to detect whether the part to be tested has offset. If offset occurs, it is confirmed that the part to be tested is a defective part. If the part to be tested has not offset, it is determined that the part to be tested is a qualified part.

[0098] In an embodiment of the present application, the parts to be tested are detected by a parts missing detection model, a parts offset detection model, and a parts defect detection model respectively, so as to determine whether the parts to be tested are missing, offset, or defective, thereby being able to comprehensively detect problems existing in the parts to be tested and improve detection efficiency and detection accuracy.

[0099] For the sake of Figure 7 The embodiment shown is described more completely below in conjunction with Figure 8 Description: When a part to be tested is transported on a conveyor belt, a target detection method is used to determine whether the part is missing based on its color. If the saturation difference between the part's color and the background color is significant, the part's missingness is determined using LAB image processing. If the saturation difference between the part's color and the background color is not significant, the part's missingness is determined using a deep learning model. If the part is determined to be missing, it is determined to be defective. If the part is not missing, the target detection method is determined by determining whether there are fixed structural components surrounding the part. If there are fixed structural components surrounding the part, the feature relationship between the part and the structural components is used to determine whether the part is offset. If there are no fixed structural components surrounding the part, the real-world coordinates of the part are obtained through bilinear interpolation. The relationship between the real-world coordinates of the part and the real-world coordinates of the reference part is used to determine whether the part is offset. If the part is offset, it is determined to be defective and detection is terminated. If the part is not offset, the part is inspected using a deep learning detection model to determine whether it is defective. If the part to be tested has defects, the part to be tested is determined to be a bad part and the test is stopped; if it is determined that the part to be tested does not have defects, the part to be tested is determined to be a qualified part and the test result of the part to be tested is output.

[0100] In the embodiment of the present application, NeNeDoggou accurately calculates the real information of the parts through feedback, ensuring that qualified precision components are provided for the product.

[0101] In some embodiments of the present application, a vision-based defective parts detection device is provided, and the vision-based defective parts detection device corresponds one-to-one with the vision-based defective parts detection method in the above-mentioned embodiment. Figure 9 As shown, the vision-based defective parts detection device includes an acquisition module 101 and a detection module 102 .

[0102] An acquisition module 101 is used to acquire a target image of a region of interest, where the region of interest includes a standard region where the part to be measured is located;

[0103] A detection module 102 is configured to detect the target image based on a part detection model to determine whether the part to be detected is a defective part;

[0104] Among them, the part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested has defects based on a deep learning model.

[0105] In some embodiments of the present application, in the above-mentioned device, when the part detection model is a part missing detection model, the detection module 102 is specifically used to determine the target detection mode of the part detection model based on the color of the part to be detected; and determine whether the part to be detected is a defective part based on the target detection mode of the part missing detection model.

[0106] In some embodiments of the present application, in the above-mentioned device, the detection module 102 is specifically used to determine that the target detection mode of the part missing detection model is the first detection mode if the saturation of the color of the part to be tested and the background color of the target image is greater than a threshold; if the saturation of the color of the part to be tested and the background color of the target image is less than or equal to a threshold, determine that the target detection mode of the part missing detection model is the second detection mode.

[0107] In some embodiments of the present application, in the above-mentioned device, when the target detection method is the first detection method, the detection module 102 is specifically used to split the target image into a LAB image; obtain the corresponding grayscale area in the LAB image through the grayscale threshold of the part to be tested; and determine whether the part to be tested is a defective part based on the grayscale area and the preset area.

[0108] In some embodiments of the present application, in the above-mentioned device, the detection module 102 is specifically used to determine that the part to be tested is not a defective part if the difference between the grayscale area and the preset area is less than or equal to the preset difference; if the difference between the grayscale area and the preset area is greater than the preset difference, determine that the part to be tested is a defective part.

[0109] In some embodiments of the present application, in the above-mentioned device, when the target detection method is the second detection method, the part missing detection model is the first deep learning model, and the detection module 102 is specifically used to detect the target image based on the first deep learning model. If the first deep learning model does not detect the part to be measured, it is determined that the part to be measured is missing; or to detect the target image based on the first deep learning model. If the first deep learning model detects the part to be measured, it is determined that the part to be measured is located in the area of ​​interest, and the outer contour of the part to be measured is obtained.

[0110] In some embodiments of the present application, in the above-mentioned device, when the part detection model is a part offset detection model, the detection module 102 is specifically used to determine the target detection method of the part offset detection model based on whether there are fixed structural parts around the part to be tested; and detect the target image based on the target detection method of the part offset detection model to determine whether the part to be tested is a defective part.

[0111] In some embodiments of the present application, in the above-mentioned device, the detection module 102 is specifically used to determine that the target detection method of the part offset detection model is the third detection method if there are fixed structural parts around the part to be tested; or to determine that the target detection method of the part offset detection model is the fourth detection method if there are no fixed structural parts around the part to be tested.

[0112] In some embodiments of the present application, in the above-mentioned device, the target detection method is the third detection method, and the detection module 102 is specifically used to obtain the outer contour and the minimum circumscribed rectangle of the part to be measured; based on the characteristic relationship between the outer contour and / or the minimum circumscribed rectangle of the part to be measured and the fixed structural part or the fixed straight line, determine whether the part to be measured is offset; based on whether the part to be measured is offset, determine whether the part to be measured is a defective part.

[0113] In some embodiments of the present application, in the above-mentioned device, the characteristic relationship includes at least one of the following: the distance between the outer contour of the part to be measured and the fixed straight line; the angle difference between the minimum circumscribed rectangle of the part to be measured and the fixed straight line; the distance between the outer contour of the part to be measured and the fixed structural part; the angle difference between the minimum circumscribed rectangle of the part to be measured and the minimum circumscribed rectangle of the fixed structural part; the occlusion relationship between the part to be measured and the fixed structural part; and the intersection relationship between the part to be measured and the fixed structural part.

[0114] In some embodiments of the present application, in the above-mentioned device, the minimum circumscribed rectangle of the part to be measured is determined based on a pre-stored calibration table.

[0115] In some embodiments of the present application, in the above-mentioned device, the target detection method is the fourth detection method, and the detection module 102 is specifically used to calculate the real-world coordinates of the part to be measured based on the image coordinates of the part to be measured and the mechanical coordinates of the camera through a pre-stored coordinate calibration table; determine whether the position of the part to be measured is offset based on the comparison of the real-world coordinates of the part to be measured with the standard world coordinates of the part to be measured; and determine whether the part to be measured is a defective part based on whether the position of the part to be measured is offset.

[0116] In some embodiments of the present application, in the above-mentioned device, the standard world coordinates of the part to be measured are determined based on a pre-stored calibration table.

[0117] In some embodiments of the present application, in the above-mentioned device, the pre-stored calibration table is generated based on the following method: obtaining the mechanical point position of the camera movement; based on the distance of the camera movement, determining the image coordinates of the checkerboard corner points where the camera movement is fixed on the current image, and obtaining the real-world coordinates of the checkerboard corner points; establishing a calibration table based on the coordinates of the mechanical point position of the camera movement, the image coordinates of the current point position, and the real world coordinates.

[0118] In some embodiments of the present application, in the above-mentioned device, when the part detection model is a part defect detection model, the part defect detection model is a second deep learning model, and the detection module 102 is specifically used to detect the target image based on the second deep learning model and output an annotated image; based on whether there are defects marked in the annotated image, determine whether the part to be tested is a defective part.

[0119] In some embodiments of the present application, in the above-mentioned device, the part detection model includes a part missing detection model, a part offset detection model, and a part defect detection model. The detection module 102 is specifically used to detect whether the part to be tested is missing based on the part missing model; if the part missing detection model detects that the part to be tested is not missing, detect whether the part to be tested is offset based on the part offset model; if the part offset model detects that the part to be tested is not offset, detect whether the part to be tested is a defective part based on the part defect detection model.

[0120] In some embodiments of the present application, in the above-mentioned apparatus, the detection module 102 is specifically configured to determine that the part to be tested is a defective part if the part offset model detects that the part to be tested is not offset and the part defect detection model detects that the part to be tested has a defect; or to determine that the part to be tested is not a defective part if the part offset model detects that the part to be tested is not offset and the part defect detection model detects that the part to be tested has no defect. Detect whether the part to be tested is a defective part based on the part defect detection model.

[0121] It should be noted that any of the above-mentioned vision-based defective parts detection devices can implement the above-mentioned vision-based defective parts detection method in a one-to-one correspondence, which will not be repeated here.

[0122] Figure 10 FIG. 1 shows a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 10 As shown, at the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for its services.

[0123] The processor, network interface, and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0124] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0125] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a vision-based defective parts detection device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the aforementioned method.

[0126] The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0127] The electronic device can execute the vision-based defective parts detection method provided in multiple embodiments of the present application, and realize a vision-based defective parts detection device in Figure 9 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0128] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple application programs, the electronic device can execute the vision-based defective part detection method provided by multiple embodiments of the present application.

[0129] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0135] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0137] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A defective parts detection method based on vision, characterized in that: The method comprises: Acquire a target image of a region of interest, wherein the region of interest includes a standard area where the part to be measured is located; Detecting the target image based on a parts detection model to determine whether the part to be detected is a defective part; Among them, the part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested has defects based on a deep learning model.

2. The method according to claim 1, characterized in that When the part detection model is a missing part detection model, detecting the target image based on the part detection model to determine whether the part to be detected is a defective part includes: Determining a target detection mode of the missing parts detection model based on the color of the part to be detected; Based on the target detection method of the part missing detection model, it is determined whether the part to be tested is a defective part.

3. The method according to claim 2, characterized in that The determining of the target detection mode of the missing parts detection model based on the color of the part to be detected includes: If the saturation of the color of the part to be tested and the background color of the target image is greater than a threshold, determining that the target detection mode of the missing parts detection model is the first detection mode; If the saturation of the color of the part to be detected and the background color of the target image is less than or equal to a threshold, it is determined that the target detection mode of the missing parts detection model is the second detection mode.

4. The method according to claim 3, characterized in that When the target detection mode is the first detection mode, the target detection mode based on the part missing detection model determines whether the part to be tested is a defective part, including: Splitting the target image into LAB images; Obtaining the corresponding grayscale area in the LAB image through the grayscale threshold of the part to be tested; Determine whether the part to be tested is a defective part based on the grayscale area and the preset area.

5. The method according to claim 4, characterized in that The determining whether the part to be tested is a defective part based on the grayscale area and the preset area includes: If the difference between the grayscale area and the preset area is less than or equal to the preset difference, it is determined that the part to be tested is not a defective part; If the difference between the grayscale area and the preset area is greater than the preset difference, it is determined that the part to be tested is a defective part.

6. The method according to claim 3, characterized in that When the target detection mode is the second detection mode, the missing parts detection model is the first deep learning model; The target detection method based on the part detection model determines whether the part to be tested is a defective part, including: Detecting the target image based on a first deep learning model, and determining that the part to be tested is missing if the first deep learning model does not detect the part to be tested; or The target image is detected based on a first deep learning model. If the first deep learning model detects a part to be measured, it is determined that the part to be measured is located within a region of interest, and an outer contour of the part to be measured is obtained.

7. The method according to claim 1, characterized in that When the part detection model is a part offset detection model, detecting the target image based on the part detection model to determine whether the part to be detected is a defective part includes: Determining a target detection mode of the part offset detection model based on whether there are fixed structural parts around the part to be tested; The target image is detected based on the target detection method of the part offset detection model to determine whether the part to be tested is a defective part.

8. A defective parts detection device based on vision, characterized in that: The device comprises: An acquisition module is used to acquire a target image of a region of interest, wherein the region of interest includes a standard area where the part to be measured is located; A detection module, configured to detect the target image based on a part detection model to determine whether the part to be detected is a defective part; Among them, the part detection model includes at least one of the following: a part missing detection model, a part offset detection model, and a part defect detection model; the part missing detection model detects whether the part to be tested is missing based on the color of the part to be tested, the part offset detection model is used to detect whether the part to be tested is offset based on the part features of the part to be tested, and the part defect detection model is used to detect whether the part to be tested has defects based on a deep learning model.

9. An electronic device comprising: processor; as well as A memory arranged to store computer-executable instructions, wherein when the instructions are executed, the processor performs the steps of the vision-based defective part detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device comprising a plurality of application programs, causes the electronic device to perform the steps of the vision-based defective part detection method according to any one of claims 1 to 7.

Images