Defect detection method for surface treatment of workpiece, control device and machine system

Abstract

The invention discloses a defect detection method for surface treatment of a workpiece, a control device and a machine system. The defect detection method comprises the following steps: acquiring a first actual image of a to-be-detected workpiece which is not subjected to surface treatment; acquiring a second actual image of the to-be-detected workpiece subjected to surface treatment; the first actual image is input into a preset first processing model, the first processing model processes the first actual image to generate a first bounding box, and an area in the first bounding box corresponds to a target area, needing surface treatment, on the to-be-detected workpiece; the first actual image and the second actual image are input into a preset second processing model, the second processing model processes the first actual image and the second actual image to generate a second bounding box, and an area in the second bounding box corresponds to an actual area, subjected to surface treatment, on the workpiece to be detected; and determining whether the surface treatment of the to-be-detected workpiece has defects or not according to the first bounding box and the second bounding box, thereby improving the detection accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of workpiece detection, in particular to a defect detection method for surface treatment of a workpiece, a control device and a machine system. BACKGROUND

[0002] In a method of coloring a workpiece, Physical Vapor Deposition (PVD) is a widely used technology. This technology has the advantages of rich process, various processing effects, and better stability and durability of PVD coating. However, during the coloring process of the workpiece, the PVD coating covers the entire surface of the workpiece, including the area that needs to be colored and the area that does not need to be colored. Therefore, it is necessary to remove the PVD coating of the area that does not need to be colored, which is called De-Physical Vapor Deposition (De-PVD) processing. During the De-PVD processing, due to the slight rotation of the clamp holding the workpiece and other reasons, the De-PVD area may be offset compared to the area that does not need to be colored. Since the color change of the workpiece caused by De-PVD processing is very insignificant, it is difficult to accurately identify whether there is an offset between the De-PVD area and the area that does not need to be colored by observing with the naked eye, so it is not possible to accurately determine whether the De-PVD area is correct, thereby resulting in low accuracy of defect detection of the workpiece De-PVD processing. SUMMARY

[0003] To solve the problem of low accuracy of defect detection of workpiece De-PVD processing, the embodiments of the present application provide a defect detection method for surface treatment of a workpiece, a control device and a machine system.

[0004] The present application provides a defect detection method for surface treatment of a workpiece. The defect detection method comprises: acquiring a first actual image of a workpiece to be detected which has not undergone surface treatment; acquiring a second actual image of the workpiece to be detected which has undergone surface treatment; inputting the first actual image into a preset first processing model, the first processing model processing the first actual image to generate a first bounding box, the area within the first bounding box corresponding to a target area on the workpiece to be detected that needs surface treatment; inputting the first actual image and the second actual image into a preset second processing model, the second processing model processing the first actual image and the second actual image to generate a second bounding box, the area within the second bounding box corresponding to an actual area on the workpiece to be detected that has undergone surface treatment; and determining whether the surface treatment of the workpiece to be detected has defects according to the first bounding box and the second bounding box.

[0005] In some embodiments, the second processing model comprises a first network branch, a second network branch, a feature fusion layer, and an output layer; and processing the first actual image and the second actual image by the second processing model to generate a second bounding box comprises: extracting features of the first actual image by the first network branch to obtain first feature data, and extracting features of the second actual image by the second network branch to obtain second feature data; performing fusion processing on the first feature data and the second feature data by the feature fusion layer to obtain fusion data; and performing restoration processing on the fusion data by the output layer to generate the second bounding box.

[0006] In some embodiments, determining whether the surface treatment of the workpiece under inspection has defects according to the first bounding box and the second bounding box comprises: superimposing the first bounding box and the second bounding box to determine whether the second bounding box exceeds the first bounding box; and in the case that the second bounding box does not exceed the first bounding box, determining that the surface treatment of the workpiece under inspection has no defects.

[0007] In some embodiments, determining whether the surface treatment of the workpiece under inspection has defects according to the first bounding box and the second bounding box further comprises: in the case that the second bounding box exceeds the first bounding box, obtaining a first size of the second bounding box exceeding the first bounding box in a first direction; in the case that the first size is greater than a preset first threshold, determining that the workpiece under inspection has defects; or in the case that the first size is less than or equal to the first threshold, determining that the workpiece under inspection has no defects.

[0008] In some embodiments, determining whether the surface treatment of the workpiece under inspection has defects according to the first bounding box and the second bounding box further comprises: in the case that the second bounding box exceeds the first bounding box, obtaining a second size of the second bounding box exceeding the first bounding box in a second direction, the first direction being perpendicular to the second direction; in the case that the second size is greater than a preset second threshold, determining that the workpiece under inspection has defects; or in the case that the second size is less than or equal to the second threshold, determining that the workpiece under inspection has no defects.

[0009] In some embodiments, the defect detection method further comprises: in the case that the first size is greater than a preset third threshold, and / or the second size is greater than a preset fourth threshold, feeding back the second bounding box to a machine performing surface treatment, the third threshold being greater than or equal to the preset first threshold, and the fourth threshold being greater than or equal to the preset second threshold.

[0010] In some embodiments, the training method of the second processing model comprises: obtaining a plurality of first training images of a plurality of training workpieces without surface treatment to form a first data set, wherein each of the plurality of first training images corresponds to one of the plurality of training workpieces; inputting the first data set into a first processing model for processing, and generating an inference bounding box in each of the first training images, wherein a region within the inference bounding box corresponds to a target region on the training workpiece that needs surface treatment; for each of the first training images, moving the inference bounding box and generating a hypothesis bounding box to form a second data set comprising a plurality of second training images; and inputting the first data set and the second data set into a second preset model for training to obtain the second processing model.

[0011] In some embodiments, the second preset model comprises a twin network model and a back propagation model, the twin network model comprises a first network branch, a second network branch, a feature fusion layer, and an output layer; the back propagation model stores a second image set, the second image set is formed by the first training images with labeled boxes, a region in the labeled box corresponds to a target region on the training workpiece that needs surface treatment; the inputting the first data set and the second data set into the second preset model for training to obtain the second processing model comprises: extracting part of the data of the first data set as a first sub-data set, and extracting part of the data of the second data set as a second sub-data set, wherein a first training image in the first sub-data set corresponds to a second training image in the second sub-data set; extracting features of the first training images in the first sub-data set using the first network branch to obtain first feature results, and extracting features of the second training images in the second sub-data set using the second network branch to obtain second feature results; performing combined processing on the first feature results and the second feature results using the feature fusion layer to obtain combined data; performing restoration processing on the combined data using the output layer to obtain third training images with training bounding boxes, and forming a third data set comprising a plurality of the third training images; comparing the third training images in the third data set with corresponding images in the second image set and calculating loss function values of each corresponding pixel; updating the twin network model according to the loss function values, and repeatedly performing the steps of extracting features and updating the twin network model using the updated twin network model; and inputting the remaining part of the data of the first data set and the remaining part of the data of the second data set into each updated twin network model, respectively, and obtaining model scores corresponding to each updated twin network model, and selecting the twin network model with the highest model score as the second processing model.

[0012] In some embodiments, the training method of the first processing model comprises: obtaining a plurality of first training images of a plurality of training workpieces without surface treatment to form a first image set, wherein the plurality of first training images correspond to the plurality of training workpieces one by one; marking a target region on the corresponding training workpiece that needs to be surface treated on each first training image to form a second image set; and inputting the second image set into a first preset model for training to obtain the first processing model.

[0013] In a second aspect, the present application provides a surface treatment control device. The control device comprises a memory and a processor. The memory is used to store instructions, and the instructions stored in the memory are executed by the processor to implement the defect detection method of any of the above embodiments.

[0014] In a third aspect, the present application provides a machine system. The machine system comprises the surface treatment control device and the surface treatment machine of any of the above embodiments. The machine is in communication connection with the control device, and is used to treat the defects by laser when the control device determines that the surface treatment of the workpiece under test has defects.

[0015] In the surface treatment defect detection method, the surface treatment control device and the machine system of the present application, the first processing model is used to process the image of the workpiece under test without surface treatment to obtain the target region of the workpiece under test that needs to be surface treated, the second processing model is used to process the image of the workpiece under test without surface treatment and the image of the workpiece under test with surface treatment to obtain the actual region of the workpiece under test with surface treatment, and finally the surface treatment of the workpiece under test is determined according to the target region and the actual region. The whole defect detection method relies on images, first processing model and second processing model, and does not need to be observed by human eyes, that is, human factors are avoided. Therefore, the surface treatment defect detection method of the present application can accurately determine whether the actual region (deplating region) of surface treatment is correct, thereby improving the accuracy and efficiency of defect detection of workpiece surface treatment.

[0016] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings, wherein:

[0018] Figure 1 is a flowchart of a surface treatment defect detection method of a workpiece according to some embodiments of the present application;

[0019] Figure 2 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0020] Figure 3 is a structural diagram of a control device according to some embodiments of the present application;

[0021] Figure 4 are partial appearance diagrams of a workpiece to be inspected without surface treatment, a workpiece to be inspected without defects in surface treatment, and a workpiece to be inspected with defects in surface treatment according to some embodiments of the present application;

[0022] Figure 5 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0023] Figure 6 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0024] Figure 7 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0025] Figure 8 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0026] Figure 9 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0027] Figure 10 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0028] Figure 11 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0029] Figure 12 is a flowchart of a method of generating a first data set and a second data set according to some embodiments of the present application;

[0030] Figure 13 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0031] Figure 14 is a flowchart of a defect detection method of surface treatment of a workpiece according to some embodiments of the present application;

[0032] Figure 15 is a flowchart of a method of generating a first sub-data set and a second sub-data set according to some embodiments of the present application;

[0033] Figure 16 is a flowchart of a defect detection method of surface treatment of a workpiece of some embodiments of the present application;

[0034] Figure 17 is a flowchart of a defect detection method of surface treatment of a workpiece of some embodiments of the present application;

[0035] Figure 18 is a flowchart of a method of training a first training model of some embodiments of the present application;

[0036] Figure 19 is a structural diagram of a machine system of some embodiments of the present application.

[0037] Reference signs in the detailed description of the embodiments are as follows:

[0038] Machine system 100;

[0039] Control device 10; memory 11; processor 13;

[0040] Machine 30. DETAILED DESCRIPTION

[0041] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a number of ways other than those described herein, and it is understood that similar improvements can be made by those skilled in the art without departing from the spirit of the present application, and therefore the present application is not limited to the specific embodiments disclosed below.

[0042] In the description of the present application, it should be understood that the terms "center", "length", "upper", "lower", "front", "rear", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0043] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features referred to. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0044] In the present application, unless specifically defined and limited otherwise, the terms "mounting", "connected", "connection" and the like should be interpreted broadly, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the internal communication of two elements or the interaction relationship between two elements, unless specifically defined otherwise. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0045] In the present application, unless specifically defined and limited otherwise, the first feature is "on" or "under" the second feature. The first and second features can be in direct contact or indirectly contact through an intermediate medium. Moreover, the first feature "above", "over" and "on" the second feature can be directly above or obliquely above the second feature, or only indicate that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "under" and "under" the second feature can be directly below or obliquely below the second feature, or only indicate that the horizontal height of the first feature is less than that of the second feature.

[0046] In the method of coloring the workpiece, physical vapor deposition (PVD) is a widely used technology. This technology has the advantages of rich process, various processing effects, and better stability and durability of PVD coating. However, during the coloring process of the workpiece, the PVD coating covers the entire surface of the workpiece, including the coloring area and the non-coloring area. Therefore, it is necessary to remove the PVD coating of the non-coloring area, which is called De-Physical Vapor Deposition (De-PVD) processing. During the De-PVD processing, due to the slight rotation of the clamp for clamping the workpiece and other reasons, the De-PVD area may be offset compared with the non-coloring area. Since the color change of the workpiece caused by De-PVD processing is very insignificant, it is difficult to accurately identify whether there is an offset between the De-PVD area and the non-coloring area by human eye observation, so it is impossible to accurately determine whether the De-PVD area is correct, thereby resulting in low accuracy of defect detection of the workpiece De-PVD processing. In order to solve the above problems, the present application provides a defect detection method for surface treatment of a workpiece (as shown in Figure 2 , a control device (as shown in Figure 3 ) and a machine system 100 (as shown in Figure 19 ).

[0047] Please refer to Figure 1 and Figure 2 , the present application provides a defect detection method for surface treatment of a workpiece. The defect detection method comprises:

[0048] 03: obtaining a first actual image of the workpiece to be inspected without surface treatment;

[0049] 04: obtaining a second actual image of the workpiece to be inspected after surface treatment;

[0050] 05: inputting the first actual image into a preset first processing model, the first processing model processing the first actual image to generate a first bounding box, the region within the first bounding box corresponding to a target region on the workpiece to be inspected that needs surface treatment;

[0051] 06: inputting the first actual image and the second actual image into a preset second processing model, the second processing model processing the first actual image and the second actual image to generate a second bounding box, the region within the second bounding box corresponding to an actual region on the workpiece to be inspected that has undergone surface treatment; and

[0052] 07: determining whether the surface treatment of the workpiece to be inspected has defects according to the first bounding box and the second bounding box.

[0053] Correspondingly, please refer to Figure 3 The present application provides a surface treatment control device 10. The control device 10 comprises a memory 11 and a processor 13. The memory 11 is used to store instructions, and the instructions stored in the memory 11 are executed by the processor 13 to realize the defect detection method in 03, 04, 05, 06 and 07. More specifically, the processor 13 is configured to: obtain a first actual image of the workpiece to be inspected without surface treatment; obtain a second actual image of the workpiece to be inspected after surface treatment; input the first actual image into a preset first processing model, the first processing model processing the first actual image to generate a first bounding box, the region within the first bounding box corresponding to a target region on the workpiece to be inspected that needs surface treatment; input the first actual image and the second actual image into a preset second processing model, the second processing model processing the first actual image and the second actual image to generate a second bounding box, the region within the second bounding box corresponding to an actual region on the workpiece to be inspected that has undergone surface treatment; and determine whether the surface treatment of the workpiece to be inspected has defects according to the first bounding box and the second bounding box.

[0054] Specifically, in the above embodiments, the surface treatment defect detection method of the workpiece is used to detect whether the surface treatment area on the workpiece after surface treatment has defects. The workpiece can be a finished product of an electronic product such as a mobile phone, a tablet computer or a notebook computer, or a semi-finished product or raw material for manufacturing an electronic product such as a mobile phone, a tablet computer or a notebook computer. The surface treatment of the workpiece can include but is not limited to grinding treatment, polishing treatment, electroplating treatment, spraying treatment and stripping treatment, etc. The above surface treatment can be performed on the entire surface of the workpiece, or only on part of the surface of the workpiece. When the actual surface treatment range deviates from the preset surface treatment range, the surface treatment of the workpiece has defects, and the defect detection method of the present application is used to determine whether the surface treatment of the workpiece to be detected has defects, i.e. whether the surface treatment area deviates from the target area that needs to be surface treated. The present application takes stripping treatment as an example of surface treatment.

[0055] The surface treatment control device 10 is a device for detecting and analyzing whether the workpiece to be detected after surface treatment has defects. The storage 11 is used to store instructions corresponding to the defect detection method, and the processor 13 is used to execute the instructions stored in the storage 11 to enable the control device 10 to implement the defect detection method. The storage 11 is electrically connected with the processor 13, and connects various parts of the entire control device 10 through various interfaces and lines. The processor 13 executes the methods in 03, 04, 05, 06 and 07 by running or loading the instructions stored in the storage 11 and calling the data stored in the storage 11, thereby realizing the defect detection.

[0056] After the workpiece to be detected is surface treated, at least part of the area is changed. Please refer to Figure 4 Before the workpiece to be detected is surface treated, the processor 13 obtains an image of the workpiece to be detected as a first actual image, and the first actual image is used to reflect the state of the workpiece to be detected before surface treatment, as shown in the left image of Figure 4 In some embodiments, the control device 10 can also include a camera, and the camera obtains the first actual image. The processor 13 of the control device 10 can directly obtain the first actual image from the camera, or the camera can store the first actual image in the storage 11, and the processor 13 obtains the first actual image from the storage 11. In other embodiments, the camera is external to the control device 10, and the first actual image obtained by the camera is stored in the cloud. The processor 13 can also obtain the first actual image stored in the cloud through communication with the cloud.

[0057] After the workpiece to be detected is surface treated, the processor 13 obtains an image of the workpiece to be detected as a second actual image, and the second image is used to reflect the state of the workpiece to be detected after surface treatment, as shown in the middle and right images of Figure 4 , wherein,Figure 4 the middle image is an image of the workpiece to be inspected without defects after surface treatment, Figure 4 the right image is an image of the workpiece to be inspected with defects after surface treatment, Figure 4 the semi-transparent white rectangle in the middle image and the right image represents the region of surface treatment. The acquisition method of the second actual image by the control device 10 can refer to the explanation of the acquisition method of the first actual image, which will not be expanded here. After the first actual image and the second actual image are acquired, the processor 13 obtains the images that can reflect the state of the workpiece to be inspected before and after surface treatment at the same time, and judges whether the surface treatment of the workpiece to be inspected has defects based on this. The information obtained after processing and analyzing the first actual image and the second actual image is more accurate in judging whether the surface treatment of the workpiece to be inspected has defects, and is easier to analyze than the information obtained only by the image of the workpiece to be inspected after surface treatment.

[0058] Please refer to Figures 1 to 3 , the first processing model is a model for judging the target region of the workpiece to be inspected that needs surface treatment according to the first actual image. The first processing model is a preset fixed model, which can be a fixed model that cannot be modified stored in the memory 11 of the control device 10 before leaving the factory, or a model written, adjusted or replaced in the memory 11 by the operator. After the first actual image is input into the preset first processing model, the first processing model processes the first actual image to generate the first bounding box. The first bounding box can be a box directly marked on the basis of the first actual image, or a box marked on the basis of the first actual image after preprocessing, wherein the preprocessing includes grayscale processing, such as Figure 2 the white box is marked on the first actual image after it is processed into all black as shown in the upper image of FIG. 1. The first bounding box can be but not limited to a rectangular box, a circular box or an irregularly shaped box, etc. The first bounding box corresponds to at least part of the region in the first actual image, which is the target region of the workpiece to be inspected that needs surface treatment obtained by the first processing model, as shown in Figure 2 It can be understood that Figure 2 the appearance of the first bounding box in FIG. 1 is only for example, and is not limited to the specific appearance of the first bounding box. In actual application, the first bounding box can be any recognizable appearance, including color, transparency, or pattern, etc. In one example, the first processing model can be a YOLO model.

[0059] The second processing model is a model for determining the actual region on which the surface treatment is performed on the workpiece to be inspected according to the first actual image and the second actual image. The second processing model is a preset fixed model, which can be a fixed model that cannot be modified and is stored in the storage 11 before the control device 10 leaves the factory, or a model that is written, adjusted or replaced in the storage 11 by an operator. After the first actual image and the second actual image are input into the preset second processing model, the second processing model processes the first actual image and the second actual image, and generates a second bounding box. The second bounding box can be a box directly marked on the second actual image, or a box marked on the second actual image after pre-processing, wherein the pre-processing includes grayscale processing, such as Figure 2 processing the second actual image into all black and then marking a gray box thereon. The second bounding box can be, but is not limited to, a rectangular box, a circular box or an irregularly shaped box, etc. The second bounding box corresponds to at least part of the region in the second actual image, which is the actual region of the workpiece to be inspected on which the surface treatment is performed by the second processing model, as shown in the lower right of Figure 2 It can be understood that Figure 2 The appearance of the second bounding box is only for example, and is not limited to the specific appearance of the second bounding box. In actual applications, the second bounding box can have any recognizable appearance, including color, transparency, or pattern, etc. In one example, the second processing model is a trained twin network model.

[0060] The processor 13 compares the first bounding box reflecting the target region of the workpiece to be inspected requiring surface treatment and the second bounding box reflecting the actual region of the workpiece to be inspected on which the surface treatment is performed, to determine whether the target region and the actual region of the workpiece to be inspected are deviated, as shown in the lower right of Figure 2 In the case that the first bounding box and the second bounding box overlap or the non-overlapping region is small, the processor 13 determines that the target region and the actual region are not deviated, and the surface treatment of the workpiece to be inspected has no defects. In the case that the first bounding box and the second bounding box do not overlap and the non-overlapping region is large, the processor 13 determines that the target region and the actual region are deviated, and the surface treatment of the workpiece to be inspected has defects.

[0061] In the surface treatment defect detection method and the surface treatment control device 10 of the present application, the first processing model is used to process the image of the workpiece to be detected without surface treatment to obtain a target region on the workpiece to be detected that needs to be subjected to surface treatment, and the second processing model is used to process the images of the workpiece to be detected without surface treatment and the workpiece to be detected subjected to surface treatment to obtain an actual region on the workpiece to be detected subjected to surface treatment, and finally, whether the surface treatment of the workpiece to be detected has defects is determined according to the target region and the actual region. The entire defect detection method relies on images, a first processing model and a second processing model, and does not need to be observed by human eyes, that is, human factors are avoided. Therefore, the surface treatment defect detection method of the present application can accurately determine whether the actual region (deplating region) of surface treatment is correct, thereby improving the accuracy of defect detection of workpiece surface treatment.

[0062] Please refer to Figure 5 and Figure 6 In some embodiments, the second processing model includes a first network branch, a second network branch, a feature fusion layer and an output layer; 06: the second processing model processes the first actual image and the second actual image to generate a second bounding box, including:

[0063] 061: using the first network branch to extract features of the first actual image to obtain first feature data, and using the second network branch to extract features of the second actual image to obtain second feature data;

[0064] 063: using the feature fusion layer to perform fusion processing on the first feature data and the second feature data to obtain fusion data; and

[0065] 065: using the output layer to perform restoration processing on the fusion data to generate a second bounding box.

[0066] Further, please refer to Figure 3 The processor 13 is further configured to perform the methods in 061, 063 and 065. Specifically, the processor 13 is configured to: use the first network branch to extract features of the first actual image to obtain first feature data, and use the second network branch to extract features of the second actual image to obtain second feature data; use the feature fusion layer to perform fusion processing on the first feature data and the second feature data to obtain fusion data; and use the output layer to perform restoration processing on the fusion data to generate a second bounding box.

[0067] Specifically, the first network branch is a part of the second processing model for extracting features from the input first actual image. The processor 13 extracts first feature data using the first network branch, and the first feature data can reflect the first actual image. The first feature data can include color information, brightness information, contrast information, texture information, edge information, shape information, line information or key point information, etc.

[0068] The second network branch is a part of the second processing model for extracting features from the input second actual image. The processor 13 extracts second feature data using the second network branch, which can reflect the second actual image. The second feature data can include color information, brightness information, contrast information, texture information, edge information, shape information, line information, or key point information, etc. The second network branch has the same structure as the first network branch and shares weights, so as to perform more targeted feature extraction.

[0069] After the processor 13 obtains the first feature data and the second feature data using the first network branch and the second network branch respectively, the two need to be processed cooperatively. The feature fusion layer is a part of the second processing model for integrating feature data. The feature fusion layer performs fusion processing on the first feature data and the second feature data, integrates complementary information in the first feature data and the second feature data, so as to improve the understanding ability of the second processing model for the input image. The feature fusion technology applied by the feature fusion layer includes but is not limited to skip connection, residual connection, feature pyramid network, attention mechanism guided fusion, multi-branch fusion, and dense connection, etc. The second processing model obtains fusion data through the feature fusion layer. The fusion data is comprehensive feature data that can reflect the first actual image and the second actual image at the same time.

[0070] The output layer is a part for converting the fusion data obtained by the feature fusion layer into the result required by the second processing model. According to the processing target of the model applied by the output layer, the output layer can output an image processing result such as labeling an image, performing a semantic segmentation task such as partitioning an image, or performing a target detection task such as labeling a target region. The output layer in the present application is used to perform a target detection task. The processor 13 restores the fusion data using the output layer to detect the position of the actual region in the second actual image and labels it through a second bounding box. In summary, the processor 13 uses the first network branch, the second network branch, the feature fusion layer, and the output layer of the second processing model to perform feature extraction, feature fusion, and feature analysis on the input first actual image and second actual image, so as to quickly and accurately generate a second bounding box that can represent the actual region, facilitating subsequent accurate analysis of whether the surface treatment of the workpiece under test has defects.

[0071] Please refer to Figure 2 and Figure 7 In some embodiments, 07: determining whether the surface treatment of the workpiece under test has defects according to the first bounding box and the second bounding box, comprises:

[0072] 071: superimposing the first bounding box and the second bounding box to determine whether the second bounding box exceeds the first bounding box; and

[0073] 073: In the case that the second bounding box does not exceed the first bounding box, it is determined that the surface treatment of the workpiece to be inspected has no defects.

[0074] Further, the processor 13 is further configured to execute the methods in 071 and 073. Specifically, the processor 13 is configured to: superimpose the first bounding box and the second bounding box, determine whether the second bounding box exceeds the first bounding box; and in the case that the second bounding box does not exceed the first bounding box, it is determined that the surface treatment of the workpiece to be inspected has no defects.

[0075] Specifically, in order to accurately determine whether the surface treatment area of the workpiece to be inspected is shifted, the parameters such as the size or the shooting angle of the first actual image and the second actual image are completely consistent, so that the size of the first bounding box and the second bounding box generated is matched, and the processing and analysis can be directly performed. At this time, the processor 13 can directly superimpose the first bounding box and the second bounding box, and then judge the relative position of the second bounding box and the first bounding box, that is, determine whether the second bounding box exceeds the first bounding box, as shown in the right middle figure of FIG. 7. Figure 2 Figure 7 As shown, the second bounding box exceeds the first bounding box. It can be understood that in the case that the second bounding box does not exceed the first bounding box, the white rectangular box coincides with the gray rectangular box.

[0076] After the first bounding box and the second bounding box are superimposed, the processor 13 detects whether the second bounding box exceeds the first bounding box. If the second bounding box does not exceed the first bounding box, it indicates that the actual area does not exceed the target area, that is, the surface treatment is not performed in the area outside the target area. At this time, the processor 13 can determine that the surface treatment of the workpiece to be inspected has no defects. Therefore, the relative position relationship between the first bounding box and the second bounding box can be used to accurately determine whether the surface treatment of the workpiece to be inspected has defects.

[0077] Please refer to Figure 2 and Figure 8 In some embodiments, 07: determining whether the surface treatment of the workpiece to be inspected has defects according to the first bounding box and the second bounding box further comprises:

[0078] 075: In the case that the second bounding box exceeds the first bounding box, a first size of the second bounding box exceeding the first bounding box in the first direction is obtained.

[0079] 077: In the case that the first size is greater than a preset first threshold, it is determined that the workpiece to be inspected has defects; or

[0080] 079: In the case that the first size is less than or equal to the first threshold, it is determined that the workpiece to be inspected has no defects.

[0081] ​Further, the processor 13 is further configured to execute the methods in 075, 077 and 079. Specifically, the processor 13 is configured to: in the case that the second bounding box exceeds the first bounding box, obtain a first size of the second bounding box exceeding the first bounding box in a first direction; in the case that the first size is greater than a preset first threshold, determine that the workpiece under inspection has a defect; and in the case that the first size is less than or equal to the first threshold, determine that the workpiece under inspection has no defect.

[0082] Specifically, the process of superimposing the first bounding box and the second bounding box is the same as described above, which will not be repeated here. After the first bounding box and the second bounding box are superimposed, the processor 13 detects whether the second bounding box exceeds the first bounding box. The second bounding box exceeding the first bounding box indicates that the actual area exceeds the target area, i.e., at least part of the area outside the target area is surface treated. Since there is an inevitable system error in mechanical processing, the actual area and the target area may deviate, but in the case that the deviation is less than the error range allowed by the surface treatment, the processor 13 can still consider that the workpiece under inspection has no defect.

[0083] Therefore, at this time, the processor 13 determines that the surface treatment of the workpiece under inspection may have a defect, and needs to be further analyzed in combination with the specific size of the second bounding box exceeding the first bounding box. According to the superimposition result of the first bounding box and the second bounding box, the processor 13 obtains the range of the second bounding box exceeding the first bounding box in the first direction, i.e., the first size. In this application, the length direction of the workpiece is defined as the first direction, and the width direction of the workpiece is defined as the second direction, and the first direction is perpendicular to the second direction. Of course, in other embodiments, the width direction of the workpiece can also be defined as the first direction, and the length direction of the workpiece can be defined as the second direction. The first direction in the first bounding box and the first direction in the second bounding box are the same direction, and at this time the processor 13 can accurately determine the first size. Alternatively, the workpiece includes two metal parts spaced apart from each other and a plastic part, the plastic part is formed by injection molding, and in the first direction, the two metal parts are connected by the plastic part. After the workpiece is subjected to PVD to form a plating layer, the plating layer on the plastic part is treated by laser De-PVD to obtain the size of the plastic part frame to obtain the second bounding box. In some examples, since the plastic part has no metal part in the second direction, all the plating layer of the plastic part in the second direction needs to be removed by De-PVD, and therefore after laser De-PVD, the plastic part in the second direction has no possibility of defect, and therefore only the size in the first direction needs to be detected.

[0084] The preset first threshold value is a maximum value of the size of the second bounding box allowed to exceed the first bounding box in the first direction. The first threshold value can be a fixed value stored in the memory 11 of the control device 10 before leaving the factory and cannot be modified, or can be a variable value set in the memory 11 by the operator during the use of the control device 10. The smaller the first threshold value, the more stringent the defect detection of the surface treatment of the workpiece to be inspected.

[0085] The processor 13 compares the first size with the first threshold value. In the case that the first size is greater than the preset first threshold value, the processor 13 determines that the size of the second bounding box exceeding the first bounding box is not allowed, and the workpiece to be inspected has defects and is a defective product. In the case that the first size is less than or equal to the first threshold value, the processor 13 determines that the size of the second bounding box exceeding the first bounding box is allowed, and the workpiece to be inspected has no defects and is a good product. Therefore, the processor 13 can determine whether the workpiece to be inspected has defects according to the first size of the second bounding box exceeding the first bounding box in the first direction and the preset first threshold value, so as to accurately determine whether the workpiece to be inspected is a defective product that needs to be eliminated.

[0086] Please refer to Figure 2 and Figure 9 In some embodiments, the workpiece includes a metal part and a plastic part formed by injection molding, and the plastic part is connected to the metal part in the first direction and the second direction. That is, the position of the laser De-PVD of the workpiece is a position other than the edge of the workpiece (for example, a middle position of the workpiece or a position close to the middle), and after the PVD is performed on the workpiece to form a coating layer, the coating layer on the plastic part is processed by laser De-PVD to obtain the second bounding box of the plastic part. In some examples, since the plastic part has a metal part in the first direction and the second direction, the coating layer on the plastic part needs to be removed by De-PVD, and therefore, the position of the De-PVD in the second direction (width direction) needs to be detected after laser De-PVD. Further, 07: determining whether the surface treatment of the workpiece to be inspected has defects according to the first bounding box and the second bounding box further includes:

[0087] 074: In the case that the second bounding box exceeds the first bounding box, obtaining a second size of the second bounding box exceeding the first bounding box in the second direction, the first direction being perpendicular to the second direction;

[0088] 076: In the case that the second size is greater than a preset second threshold value, determining that the workpiece to be inspected has defects; or

[0089] 078: In the case that the second size is less than or equal to the second threshold value, determining that the workpiece to be inspected has no defects.

[0090] Further, the processor 13 is also configured to execute the methods in 074, 076 and 078. Specifically, the processor 13 is configured to: in the case that the second bounding box exceeds the first bounding box, obtain a second size of the second bounding box exceeding the first bounding box in a second direction, the first direction being perpendicular to the second direction; in the case that the second size is greater than a preset second threshold, determine that the workpiece under inspection has defects; and in the case that the second size is less than or equal to the second threshold, determine that the workpiece under inspection has no defects.

[0091] Specifically, the process of superimposing the first bounding box and the second bounding box is the same as above, which will not be repeated here. After superimposing the first bounding box and the second bounding box, the processor 13 detects whether there is an area of the second bounding box exceeding the first bounding box. The second bounding box exceeding the first bounding box indicates that the actual area exceeds the target area, i.e., at least part of the area outside the target area is surface treated. Since there is an unavoidable system error in mechanical processing, in the case that the actual area deviates from the target area but the deviation degree is less than the error range allowed by surface treatment, the processor 13 can still consider that the workpiece under inspection has no defects.

[0092] Therefore, at this time, the processor 13 determines that the surface treatment of the workpiece under inspection may have defects, and needs to be further analyzed in combination with the specific size of the second bounding box exceeding the first bounding box. According to the superimposition result of the first bounding box and the second bounding box, the processor 13 obtains the range of the second bounding box exceeding the first bounding box in the second direction, i.e., the second size. The second direction in the first bounding box and the second direction in the second bounding box are the same direction, and at this time the processor 13 can accurately determine the second size.

[0093] The preset second threshold is the maximum value of the size of the second bounding box exceeding the first bounding box in the second direction. The second threshold can be an unmodifiable fixed value stored in the storage 11 by the control device 10 before leaving the factory, or a variable value that can be set in the storage 11 by the operator during the use of the control device 10. The smaller the second threshold is, the more stringent the defect detection of the surface treatment of the workpiece under inspection is.

[0094] The processor 13 compares the second size with the second threshold, and in the case that the second size is greater than the preset second threshold, the processor 13 determines that the size of the second bounding box exceeding the first bounding box is not allowed, and the workpiece under inspection has defects, which is a defective product. In the case that the second size is less than or equal to the second threshold, the processor 13 determines that the size of the second bounding box exceeding the first bounding box is allowed, and the workpiece under inspection has no defects, which is a good product. Therefore, the processor 13 can determine whether the workpiece under inspection has defects according to the second size of the second bounding box exceeding the first bounding box in the first direction and the preset second threshold, so as to accurately determine whether the workpiece under inspection is a defective product that needs to be eliminated.

[0095] Please refer toFigure 2 and Figure 10 In some embodiments, the defect detection method further comprises:

[0096] 09: feeding back the second bounding box to a machine performing the surface treatment, in a case that the first size is greater than a preset third threshold value, and / or the second size is greater than a preset fourth threshold value, the third threshold value being greater than or equal to the preset first threshold value, and the fourth threshold value being greater than or equal to the preset second threshold value.

[0097] Further, the processor 13 is further configured to perform the method in 09. Specifically, the processor 13 is configured to: feed back the second bounding box to a machine performing the surface treatment, in a case that the first size is greater than a preset third threshold value, and / or the second size is greater than a preset fourth threshold value.

[0098] Specifically, the first size and the second size are used to represent the specific amount that the actual area exceeds the target area, corresponding to the size of the area outside the target area that has been subjected to the surface treatment. The cause of the deviation can be the slight rotation of the clamp clamping the workpiece to be inspected, the light path offset of the laser of the surface treatment device, or the change of other factors in the processing environment, etc. If the first size and the second size are too large, the machine performing the surface treatment operation cannot perform the surface treatment operation well, at which time the processor 13 needs to feed back the deviation information to the machine, so that the operator adjusts the machine performing the surface treatment, thereby reducing the defective rate of the workpiece to be inspected in subsequent processing of the machine.

[0099] The preset third threshold value is the value of the first size corresponding to the limit value of the defective rate of the workpiece to be inspected allowed to be processed by the machine. The third threshold value can be a fixed value that cannot be modified and is stored in the storage 11 by the control device 10 before leaving the factory, or can be a variable value that can be set in the storage 11 by the operator during the use of the control device 10. The smaller the third threshold value, the higher the standard of the surface treatment operation of the machine on the workpiece to be inspected, and the lower the defective rate of the workpiece to be inspected in subsequent processing. It can be understood that the third threshold value is greater than or equal to the first threshold value.

[0100] The processor 13 compares the first size with the third threshold value, and in a case that the first size is greater than the preset third threshold value, the processor 13 determines that the size of the second bounding box exceeding the first bounding box is too large, and the accuracy of the process of the machine performing the surface treatment on the workpiece to be inspected is insufficient. At this time, the processor 13 feeds back the specific information of the second bounding box to the machine performing the surface treatment, so that the machine adjusts the parameters for the surface treatment, such as the laser emission angle of the laser, the laser emission power, the installation position of the workpiece to be inspected on the machine, etc., to improve the accuracy of the machine in subsequent processing of the workpiece to be inspected.

[0101] The fourth threshold value is a value of the second size corresponding to a limit value of the defective rate of the workpiece allowed to be processed by the machine. The fourth threshold value can be a fixed value stored in the memory 11 of the control device 10 before the control device 10 is shipped, which cannot be modified, or a variable value set in the memory 11 by an operator during the use of the control device 10. The smaller the fourth threshold value is, the higher the standard of the surface treatment operation of the workpiece is, and the lower the defective rate of the workpiece processed by the subsequent machine is. It can be understood that the fourth threshold value is greater than or equal to the second threshold value.

[0102] The processor 13 compares the second size with the fourth threshold value. When the second size is greater than the preset fourth threshold value, the processor 13 determines that the second bounding box is too large to exceed the first bounding box, and the accuracy of the process of the machine performing the surface treatment on the workpiece is insufficient. At this time, the processor 13 feeds back the specific information of the second bounding box to the machine performing the surface treatment, so that the machine adjusts the parameters of the surface treatment, such as the laser emission angle of the laser, the laser emission power, the installation position of the workpiece on the machine, etc., to improve the accuracy of the subsequent processing of the workpiece by the machine.

[0103] Therefore, the processor 13 can determine whether the process of the machine performing the surface treatment on the workpiece has the problem of insufficient accuracy according to the relationship between the first size and the third threshold value, and / or the relationship between the second size and the fourth threshold value, so as to adjust the machine according to the actual deviation, so that the subsequent surface treatment of the workpiece is more accurate, and the defective rate of the processed workpiece is reduced.

[0104] Please refer to Figure 11 and Figure 12 In some embodiments, the training method of the second processing model comprises:

[0105] 021: Obtain a plurality of first training images of a plurality of training workpieces that have not undergone surface treatment to form a first data set, wherein the plurality of first training images correspond one-to-one to the plurality of training workpieces;

[0106] 023: input the first data set into the first processing model for processing, and generate an inference bounding box in each first training image, wherein the region in the inference bounding box corresponds to the target region on the training workpiece that needs surface treatment;

[0107] 025: for each first training image, move the inference bounding box and generate a hypothesis bounding box to form a second data set comprising a plurality of second training images; and

[0108] 027: input the first data set and the second data set into the second preset model for training to obtain the second processing model.

[0109] Further, the processor 13 is further configured to execute the method in 021, 023, 025 and 027. Specifically, the processor 13 is configured to: acquire a plurality of first training images of a plurality of training workpieces without surface treatment to form a first data set, wherein the plurality of first training images correspond to the plurality of training workpieces one by one; input the first data set into a first processing model for processing, and generate an inference bounding box in each first training image, wherein a region within the inference bounding box corresponds to a target region on the training workpiece that needs surface treatment; for each first training image, move the inference bounding box and generate a hypothesis bounding box to form a second data set comprising a plurality of second training images; and input the first data set and the second data set into a second preset model for training to obtain a second processing model.

[0110] Specifically, in order to enable the second processing model to effectively process the first actual image and the second actual image, an image of the same type as the first actual image and the second actual image is used for model training to obtain the second processing model.

[0111] In the process of training the second processing model, workpieces of the same type as the workpiece to be inspected are used as training workpieces, Figure 12 W1-1, W1-2, W1-3 and W1-n in the above formula represent training workpieces. The number of training workpieces is a plurality, and the plurality herein refers to more than one, i.e. two, three, four, five or six, etc. The number of training workpieces is a plurality, which can make the trained second processing model have better robustness.

[0112] The processor 13 acquires a plurality of images of the workpieces without surface treatment as first training images, and the first training images are images for indicating target regions on the inference training workpieces that need surface treatment. The set formed by all the first training images is the first data set, and in the first data set, any one first training image (similar to the aforementioned first actual image actually acquired) has a training workpiece corresponding thereto. Figure 12 P1-1, P1-2, P1-3 and P1-n in the above formula represent first training images, wherein P1-1 corresponds to the training workpiece W1-1, P1-2 corresponds to the training workpiece W1-2, P1-3 corresponds to the training workpiece W1-3, and P1-n corresponds to the training workpiece W1-n.

[0113] After the processor 13 inputs the first data set into the preset first processing model, the first processing model processes each first training image in the first data set and generates an inference bounding box corresponding to each first training image. The inference bounding box can be a box directly marked on the first training image, or a box marked on the first training image after preprocessing, wherein the preprocessing includes grayscale processing, such asFigure 12 The first training image shown in the lower left of FIG. 1 is processed into a full gray image, and a dashed line frame is marked on it. The inference bounding box can be, but is not limited to, a rectangular frame, a circular frame, or an irregular frame, etc. The inference bounding box corresponds to at least a part of the region in the first training image, which is the target region on the workpiece that needs surface treatment obtained by the first processing model.

[0114] Further, the processor 13 processes each inference bounding box to change the position of the inference bounding box. The processor 13 can change the position of the inference bounding box in the following ways, but is not limited to: moving the inference bounding box to the left, moving the inference bounding box to the right, moving the inference bounding box up, or moving the inference bounding box down. The region inside the inference bounding box is changed to a semi-transparent black region. At this time, the processor 13 takes the changed bounding box as a hypothetical bounding box, and the combination of the hypothetical bounding box and the original first training image is called a second training image. The semi-transparent black hypothetical bounding box is used to simulate the appearance of the workpiece after surface treatment, and therefore, the second training image is used to simulate the image of the training workpiece after surface treatment (similar to the aforementioned second actual image actually obtained). The set of second training images is called a second data set. The second training set is an image set used to train the second processing model together with the first training set. Figure 12 P2-1, P2-2, P2-3, and P2-n in FIG. 2 represent the second training image, where P2-1 corresponds to P1-1 and W-1, P2-2 corresponds to P1-2 and W-2, P2-3 corresponds to P1-3 and W-3, and P2-n corresponds to P1-n and W-n.

[0115] The second preset model is a model used to identify the region of the workpiece surface that has actually been treated. The second preset model is a preset fixed model, which can be a fixed model that cannot be modified and is stored in the memory 11 before the control device 10 is shipped, or a model that can be written, adjusted, or replaced in the memory 11 by the operator during use of the control device 10. Further, the processor 13 inputs the first data set and the second data set into the second preset model for training. The trained second preset model is taken as the second processing model, and the user subsequently performs defect detection on the surface treatment of the workpiece to be inspected based on the first actual image and the second actual image.

[0116] In summary, in the process of training the second processing model, the processor 13 utilizes the first data set and the second data set required for training the model with the same type of workpiece as the workpiece to be inspected. The first data set is composed of first training images used to obtain the target area on each training workpiece that needs surface treatment, i.e., corresponding to the first actual image input into the second processing model. The second data set is composed of second training images used to simulate the appearance of each training workpiece after surface treatment, i.e., corresponding to the second actual image input into the second processing model. The first data set and the second data set are input into the second preset model for training to obtain the second processing model that can better predict the actual area.

[0117] Reference may be made to Figure 14 In some embodiments, the second preset model includes a twin network model and a back propagation model, the twin network model includes a first network branch, a second network branch, a feature fusion layer, and an output layer; the back propagation model stores a second image set, the second image set is composed of first training images with a marked box, the area in the marked box corresponds to the target area on the training workpiece that needs to be surface treated. Please refer to Figure 13 027: inputting the first data set and the second data set into the second preset model for training to obtain the second processing model, comprising:

[0118] 0271: extracting part of the data in the first data set as a first sub-data set, and extracting part of the data in the second data set as a second sub-data set, the first training image in the first sub-data set corresponds to the second training image in the second sub-data set;

[0119] 0272: extracting the features of the first training image in the first sub-data set using the first network branch to obtain a first feature result, and extracting the features of the second training image in the second sub-data set using the second network branch to obtain a second feature result;

[0120] 0273: merging and processing the first feature result and the second feature result using the feature fusion layer to obtain merged data;

[0121] 0274: restoring the merged data using the output layer to obtain third training images with training bounding boxes, and composing a third data set containing multiple third training images;

[0122] 0275: comparing the third training images in the third data set with the corresponding images in the second image set and calculating the loss function value of each corresponding pixel;

[0123] 0276: updating the Siamese network model according to the loss function value, and repeatedly performing the steps of extracting features to updating the Siamese network model by using the updated Siamese network model; and

[0124] 0277: inputting the remaining part of the first data set and the remaining part of the second data set into each updated Siamese network model respectively, obtaining the model scores corresponding to each updated Siamese network model, and selecting the Siamese network model with the highest model score as the second processing model.

[0125] Further, referring to Figure 3 and Figure 13 , the processor 13 is further configured to perform the methods in 0271, 0272, 0273, 0274, 0275, 0276 and 0277. Specifically, the processor 13 is configured to: extract part of the first data set as a first sub-data set, extract part of the second data set as a second sub-data set, the first training images in the first sub-data set correspond to the second training images in the second sub-data set; extract features of the first training images in the first sub-data set by using the first network branch to obtain first feature results, and extract features of the second training images in the second sub-data set by using the second network branch to obtain second feature results; perform merging processing on the first feature results and the second feature results by using the feature fusion layer to obtain merged data; perform restoration processing on the merged data by using the output layer to obtain third training images with training bounding boxes, and form a third data set containing multiple third training images; compare the third training images in the third data set with the corresponding images in the second image set and calculate the loss function value of each corresponding pixel; update the Siamese network model according to the loss function value, and repeatedly perform the steps of extracting features to updating the Siamese network model by using the updated Siamese network model; and input the remaining part of the first data set and the remaining part of the second data set into each updated Siamese network model respectively, obtain the model scores corresponding to each updated Siamese network model, and select the Siamese network model with the highest model score as the second processing model.

[0126] Specifically, referring to Figure 14The twin network model is a feature comparison model based on deep learning. The twin network model can extract features from input samples and process the features to obtain an output result related to the fused features. The first network branch is a part of the twin network model for extracting features from one input sample. The second network branch is a part of the twin network model for extracting features from another input sample. The first network branch and the second network branch have the same structure and share weights, so as to perform more targeted feature extraction. The feature fusion layer is a part of the twin network model for integrating feature data. The feature fusion layer fuses the features obtained by the first network branch and the features obtained by the second network branch and obtains processed merged data. The output layer is a part for converting the merged data obtained by the feature fusion layer into a result required by the twin network model. According to the application scenario of the twin network model, the output layer can output different results.

[0127] The back propagation model is a model for correcting a model. The back propagation model stores an ideal sample representing an ideal result. The back propagation model compares the output result of a corrected model (such as the twin network model in the present application) with the ideal sample to determine the degree of deviation of the output result of the corrected model, and then adjusts the corrected model based on the deviation.

[0128] Referring to Figure 15 In the present application, the processor 13 randomly extracts part of the first data set, and the set of the extracted first training images constitutes a first sub-data set. The processor 13 also randomly extracts part of the second data set, and the set of the extracted second training images constitutes a second sub-data set. As shown in Figure 15 P1-i, P1-j, P1-k and P1-l represent the extracted first training images, and P2-i, P2-j, P2-k and P2-l represent the extracted second training images. P1-i corresponds to P2-i and W-i, P1-j corresponds to P2-j and W-j, P1-k corresponds to P2-k and W-k, and P1-l corresponds to P2-l and W-l. At this time, each first training image in the first sub-data set has a second training image corresponding thereto uniquely in the second sub-data set.

[0129] Referring to Figure 14 The first network branch is a part of the twin network model for extracting features from the first actual images in the input first sub-data set. The processor 13 extracts a first feature result by using the first network branch. The first feature result can reflect the first actual images in the first sub-data set, and the first feature result can include color information, brightness information, contrast information, texture information, edge information, shape information, line information or key point information, etc.

[0130] The second network branch is a part of the twin network model for extracting features from the second actual image in the second sub-data set. The processor 13 extracts a second feature result using the second network branch, which can reflect the second actual image in the second sub-data set. The second feature result can include color information, brightness information, contrast information, texture information, edge information, shape information, line information, or key point information, etc. The second network branch has the same structure as the first network branch and shares the weights to perform more targeted feature extraction.

[0131] After the processor 13 obtains the first feature result and the second feature result using the first network branch and the second network branch respectively, the two results need to be processed cooperatively. The feature fusion layer is a part of the twin network model for integrating feature results. The feature fusion layer fuses the first feature result and the second feature result, integrates complementary information in the first feature result and the second feature result, and improves the understanding ability of the twin network model for the input image. The feature fusion technology applied by the feature fusion layer includes but is not limited to skip connection, residual connection, feature pyramid network, attention mechanism guided fusion, multi-branch fusion, and dense connection, etc. The twin network model obtains merged data through the feature fusion layer. The merged data is a comprehensive feature result reflecting the first training image in the first sub-data set and the second training image in the second sub-data set.

[0132] The output layer is a part for converting the merged data obtained by the feature fusion layer into the result required by the twin network model. According to the processing target of the model applied by the output layer, the output layer can output an image processing result such as labeling an image, performing a semantic segmentation task such as partitioning an image, or performing a target detection task such as labeling a target region. The output layer in this application is used to perform a target detection task. The processor 13 restores the merged data using the output layer to detect the position (position for simulating surface processing) of the second training image in the second sub-data set corresponding to the hypothesis bounding box, and labels it through the training bounding box. The training bounding box can be a box directly labeled on the basis of the second training image in the second sub-data set, or a box labeled on the basis of the second training image after pre-processing. The second training image in the second sub-data set with the training bounding box is the third training image, and the set of third training images is the third image set.

[0133] The second image set is stored in the back propagation model, and the second image set is composed of the first training images with the marked bounding boxes (i.e., the marked bounding boxes will be introduced below), and the area in the marked bounding box corresponds to the target area on the workpiece that needs to be surface treated. Therefore, the processor 13 compares the third training image with the corresponding first training image with the marked bounding box in the second image set to determine whether the prediction result of the twin network model is accurate. Specifically, the difference between any corresponding first training image and the third training image can reflect whether the prediction result of the twin network model is accurate, and therefore, the loss function obtains the loss function value by calculating the difference between the corresponding pixels of the first training image and the third training image, and reflects the difference between the first training image and the third training image through the loss function value. Specifically, the processor 13 can normalize the pixel values or gray values of the first training image and the third training image, and then calculate the cross-entropy, mean square error, structural similarity index or Euclidean distance of each corresponding pixel of the processed first training image and the third training image as the loss function value.

[0134] The processor 13 updates the twin network model according to the loss function value by using the back propagation model, such as adjusting the structure or parameters of the twin network model, to obtain a new twin network model that is at least partially different from the twin network model before the update. On the basis of the new twin network model, the processor 13 continues to perform each step in 0271-0271, i.e., repeatedly performs the steps of extracting features to updating the twin network model, to obtain multiple twin network models. There are at least two models that are not completely the same in all the above-mentioned twin network models. The number of repeated executions can be set, and the higher the accuracy requirement of the surface treatment defect detection, the more the number of repeated executions, so as to obtain more twin network models that can be compared and selected by the processor 13.

[0135] Please refer to Figure 16 , the processor 13 randomly extracts part of the data of the first data set, and the remaining part of the first training image. Corresponding Figure 15 , the n first training images P1-1 to P1-n, except for P1-i, P1-j, P1-k and P1-l, etc. selected. The processor 13 randomly extracts part of the data of the second data set, and the remaining part of the second training image. Corresponding Figure 15 , the n first training images P2-1 to P2-n, except for P2-i, P2-j, P2-k and P2-l, etc. selected. At this time, this part of the remaining images are still one-to-one corresponding.

[0136] The processor 13 inputs the remaining part of the first data set and the remaining part of the second data set into each of the plurality of updated twin network models in turn, and generates a corresponding third training image. It can be understood that the second image set stored in the back propagation model has a corresponding first training image with a marked bounding box for each third training image. The processor 13 compares each third training image with the corresponding first training image with a marked bounding box, and obtains a numerical value representing the advantages and disadvantages of the updated twin network model, i.e. a model score, according to the result. The calculation of the model score needs to comprehensively consider the first training images with marked bounding boxes corresponding to all third training images, so as to more accurately reflect the advantages and disadvantages of the twin network model. The model score algorithm of the present application takes F1-score function as an example, which is specifically:

[0137]

[0138] wherein, Precision is the ratio of the number of overlapping pixels between the training bounding box and the marked bounding box to the training bounding box, indicating the proportion of actual positive examples in all predicted positive examples (i.e. the area that needs surface treatment after surface treatment). Recall is the ratio of the number of overlapping pixels between the training bounding box and the marked bounding box to the marked bounding box, indicating the proportion of correctly predicted positive examples in all actual positive examples. The summary of F1 obtained for each third training image is the model score, at this time, the model score can be the average or total of each F1, etc. The processor 13 determines the twin network model with the highest model score as the optimal twin network model, and uses it as the second processing model. At this time, the first network branch of the twin network model is the first network branch of the second processing model in the above, the second network branch of the twin network model is the second network branch of the second processing model in the above, the feature fusion layer of the twin network model is the feature fusion layer of the second processing model in the above, and the output layer of the twin network model is the output layer of the second processing model in the above.

[0139] In summary, in the training process of the second processing model of the present application, part of the first data set and the second data set are input into the twin network model to obtain a third training image with a training bounding box, the third training image with the training bounding box is compared with the first training image with the labeled bounding box in the second image set stored in the back propagation model, and the twin network model is updated according to the result. After multiple executions, multiple twin network models are obtained. Further, the processor 13 inputs the remaining part of the first data set and the second data set participating in the model training into each updated twin network model, respectively, to select the twin network model with the highest model score as the second processing model. At this time, the data amount used in the training process of the second processing model is large and rich, and the second processing model can better predict the actual area of the surface treatment process.

[0140] Reference is made to Figure 17 In some embodiments, the training method of the first processing model comprises:

[0141] 011: obtaining a plurality of first training images of a plurality of training workpieces without surface treatment to form a first image set, wherein the plurality of first training images correspond to the plurality of training workpieces one by one;

[0142] 013: marking the target area of the corresponding training workpiece that needs to be treated on each first training image to form a second image set; and

[0143] 015: inputting the second image set into the first preset model for training to obtain the first processing model.

[0144] Further, the processor 13 is also used to execute the method in 011, 013 and 015. Specifically, the processor 13 is configured to: obtain a plurality of first training images of a plurality of training workpieces without surface treatment to form a first image set, wherein the plurality of first training images correspond to the plurality of training workpieces one by one; mark the target area of the corresponding training workpiece that needs to be treated on each first training image to form a second image set; and input the second image set into the first preset model for training to obtain the first processing model.

[0145] Specifically, in order to enable the first processing model to effectively process the first actual image, the same type of image as the first actual image is used for model training to obtain the first processing model. In the training process of obtaining the first processing model, the same type of workpiece as the workpiece to be detected is used as the training workpiece. Here, the training workpiece is the same as the training workpiece in 021, which is represented as W3-1, W3-2, W3-3 and W3-n in the training workpiece in the first processing model. Figure 18

[0146] Reference is made to​Figure 18 The processor 13 acquires images of a plurality of training workpieces without surface treatment as first training images, and the first training images are images for inferring target regions on the training workpieces that require surface treatment. A set composed of all the first training images is a first image set, and the first image set is a first data set. In the first image set, each first training image has a training workpiece corresponding thereto. Figure 18 P’1-1, P1’-2, P1’-3, and P1-n in FIG. 1 represent first training images, wherein P1’-1 corresponds to P1-1 and W3-1, P1’-2 corresponds to P1-2 and W-2, P1’-3 corresponds to P1-3 and W3-3, and P1’-n corresponds to P1-n and W3-n.

[0147] On each first training image, the processor 13 marks a target region on the corresponding training workpiece that requires surface treatment, such as a marked bounding box (i.e., the marked box in the above) shown in FIG. 2. Figure 18 The marked bounding box can be a box directly marked on the first training image, or a box marked on a first training image after preprocessing, wherein the preprocessing includes grayscale processing. The formation of the marked bounding box can be manually marked by an operator, or automatically marked by the processor 13 through other models with marking functions. The more and more accurate the marking points are when marking the region of the marked bounding box, the more accurately the trained first processing model can detect the true position of the first target region. The set of first training images with marked bounding boxes is a second image set.

[0148] The first preset model is a model for identifying target regions on the surface of a workpiece (here, a training workpiece) that require surface treatment. The first preset model is a preset fixed model, which can be a fixed model that cannot be modified and is stored in the storage 11 before the control device 10 is shipped, or a model that can be written, adjusted, or replaced in the storage 11 by an operator during use of the control device 10. The first preset model can be any model capable of extracting a specific region from an input image, such as a YOLO model, a Mask R-CNN model, or a Mask2Former model. The processor 13 inputs the second image set into the first preset model for training, and feeds back the training result to the trained first preset model until a trained first preset model that can generate a first bounding box according to an input first actual image is obtained, and the trained first preset model is used as a first processing model.

[0149] In summary, in the process of training the first processing model, the first image set required for model training is obtained by using the same type of training workpiece as the workpiece to be inspected. The first data set is composed of first training images, and the first training images are used to display the appearance of the training workpiece without surface treatment, i.e., corresponding to the first actual image input into the second processing model. The second image set is composed of labeled first training images, and the second image set is used to provide information of target positions for the first preset model. The first image set and the second image set are input into the first preset model for training to obtain the first processing model.

[0150] Please refer to Figure 19 The present application provides a machine system 100. The machine system 100 comprises the surface treatment control device 10 and the surface treatment machine 30 of any of the above embodiments. The machine 30 is in communication connection with the control device 10, and is used to process defects by laser when the control device 10 determines that the surface treatment of the workpiece to be inspected has defects.

[0151] Specifically, the machine system 100 is a system for processing the workpiece to be inspected with defects in surface treatment. The machine system 100 comprises a control device 10 and a machine 30. The control device 10 executes the defect detection method of surface treatment in any of the above embodiments through the memory 11 and the processor 13. The machine 30 is a device for performing reprocessing on the workpiece to be inspected with defects in surface treatment. The machine 30 is in communication connection with the control device 10 so that the machine 30 can receive information from the control device 10, i.e., specific information of the workpiece to be inspected. In the case where the control device 10 determines that the surface treatment of the workpiece to be inspected has defects, the control device 10 transmits this information to the machine 30. The machine 30 determines the specific position of the defects, for example, the part of the first bounding box that does not overlap with the second bounding box represents the area that needs to be surface treated but is not surface treated. Therefore, the machine 30 can reprocess the workpiece to be inspected according to the size and position of the area, and process the defects of the part of the workpiece to be inspected corresponding to the part of the first bounding box that does not overlap with the second bounding box by laser, so as to ensure that the entire area that needs to be processed is surface treated.

[0152] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure. Meanwhile, other embodiments can be derived from the above-described embodiments, so that structural and logical substitutions and changes can be made without departing from the scope of the present disclosure.

[0153] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for detecting defects in the surface treatment of a workpiece, characterized in that, include: Obtain the first actual image of the workpiece to be inspected without surface treatment; Acquire a second actual image of the workpiece to be inspected after surface treatment; The first actual image is input into a preset first processing model. The first processing model processes the first actual image to generate a first bounding box. The area within the first bounding box corresponds to the target area on the workpiece to be inspected that requires surface treatment. The first actual image and the second actual image are input into a preset second processing model. The second processing model processes the first actual image and the second actual image to generate a second bounding box. The area within the second bounding box corresponds to the actual area on the workpiece to be inspected that has undergone surface treatment. and The presence of surface treatment defects in the workpiece to be inspected is determined based on the first bounding box and the second bounding box.

2. The defect detection method according to claim 1, characterized in that, The second processing model includes a first network branch, a second network branch, a feature fusion layer, and an output layer; The second processing model processes the first actual image and the second actual image to generate a second bounding box, including: The first network branch is used to extract features from the first actual image to obtain first feature data, and the second network branch is used to extract features from the second actual image to obtain second feature data; The first feature data and the second feature data are fused using the feature fusion layer to obtain fused data; and The output layer is used to restore the fused data to generate the second bounding box.

3. The defect detection method according to claim 1, characterized in that, The step of determining whether the surface treatment of the workpiece to be inspected has defects based on the first bounding box and the second bounding box includes: Overlay the first bounding box and the second bounding box to determine whether the second bounding box extends beyond the first bounding box; and If the second boundary frame does not exceed the first boundary frame, then it is determined that the surface treatment of the workpiece to be inspected is free of defects.

4. The defect detection method according to claim 3, characterized in that, The step of determining whether the surface treatment of the workpiece to be inspected has defects based on the first bounding box and the second bounding box further includes: If the second bounding box extends beyond the first bounding box, obtain the first dimension by which the second bounding box extends relative to the first bounding box in the first direction; If the first size is greater than a preset first threshold, it is determined that the workpiece to be inspected has a defect; or If the first size is less than or equal to the first threshold, it is determined that the workpiece to be inspected has no defects.

5. The defect detection method according to claim 4, characterized in that, The step of determining whether the surface treatment of the workpiece to be inspected has defects based on the first bounding box and the second bounding box further includes: When the second bounding box extends beyond the first bounding box, obtain a second dimension by which the second bounding box extends relative to the first bounding box in a second direction, wherein the first direction is perpendicular to the second direction; If the second dimension is greater than a preset second threshold, it is determined that the workpiece to be inspected has a defect; or If the second dimension is less than or equal to the second threshold, it is determined that the workpiece to be inspected has no defects.

6. The defect detection method according to claim 5, characterized in that, The defect detection method further includes: If the first size is greater than a preset third threshold and / or the second size is greater than a preset fourth threshold, the second bounding box is fed back to the machine performing the surface treatment, wherein the third threshold is greater than or equal to a preset first threshold and the fourth threshold is greater than or equal to a preset second threshold.

7. The defect detection method according to claim 1, characterized in that, The training methods for the second processing model include: Multiple first training images of multiple training workpieces that have not undergone surface treatment are acquired to form a first dataset, wherein the multiple first training images correspond one-to-one with the multiple training workpieces. The first dataset is input into the first processing model for processing, and an inference bounding box is generated for each of the first training images. The region within the inference bounding box corresponds to the target region on the training workpiece that needs surface treatment. For each of the first training images, the inference bounding box is moved and a hypothesis bounding box is generated to form a second dataset containing multiple second training images; and The first dataset and the second dataset are input into the second preset model for training to obtain the second processing model.

8. The defect detection method according to claim 7, characterized in that, The second preset model includes a Siamese network model and a backpropagation model. The Siamese network model includes a first network branch, a second network branch, a feature fusion layer, and an output layer. The backpropagation model stores a second image set, which consists of the first training images with bounding boxes. The regions within the bounding boxes correspond to the target regions on the training workpiece that require surface treatment. 027: The step of inputting the first dataset and the second dataset into a second preset model for training to obtain the second processing model includes: A portion of the data from the first dataset is extracted as a first subset, and a portion of the data from the second dataset is extracted as a second subset. The first training image in the first subset corresponds to the second training image in the second subset. The first feature result is obtained by extracting features of the first training image in the first subset of the dataset using the first network branch, and the second feature result is obtained by extracting features of the second training image in the second subset of the dataset using the second network branch. The first feature result and the second feature result are merged using the feature fusion layer to obtain merged data; The output layer is used to restore the merged data to obtain a third training image with training bounding boxes, and a third dataset containing multiple third training images is formed. The third training image in the third dataset is compared with the corresponding image in the second image set, and the loss function value of each corresponding pixel is calculated. The Siamese network model is updated based on the loss function value, and the step of extracting features is repeated until the step of updating the Siamese network model is performed using the updated Siamese network model; and The remaining data from the first dataset and the remaining data from the second dataset are respectively input into the Siamese network model for each update, and the model score corresponding to the Siamese network model after each update is obtained. The Siamese network model with the highest model score is selected as the second processing model.

9. The defect detection method according to claim 7, characterized in that, The training method for the first processing model includes: Multiple first training images of multiple training workpieces that have not undergone surface treatment are acquired to form a first image set, wherein the multiple first training images correspond one-to-one with the multiple training workpieces. Mark the target areas on the training workpiece that require surface treatment on each of the first training images to form a second image set; and The second image set is input into the first preset model for training to obtain the first processing model.

10. A surface treatment control device, comprising a memory and a processor, wherein the memory is used to store instructions, characterized in that, The instructions stored in the memory are executed by the processor to implement the defect detection method according to any one of claims 1-9.

11. A machine tool system, characterized in that, include: The surface treatment control device according to claim 10; and The surface treatment machine is communicatively connected to the control device and is used to treat the defects by laser when the control device determines that there are defects in the surface treatment of the workpiece to be inspected.

Images