Method and apparatus for automated optical detection

Abstract

The present application relates to a method for AOI, comprising: obtaining an image of a detected component of a first domain, the first domain corresponding to a first industrial component production line; generating, by a general feature extraction sub-model of a detection model, a first feature representation of the image based on the image of the detected component, the detection model including the general feature extraction sub-model and a plurality of task detection sub-models related to a specific AOI task of the first domain; generating, by the plurality of task detection sub-models, a plurality of detection results related to a specific AOI task of the first domain based on the first feature representation of the image; and determining the state of the detected component based on the plurality of detection results.

Description

Technical Field

[0001] The present application relates to optical inspection, and more particularly, to a method and apparatus for automatic optical inspection (AOI) of industrial components. Background Art

[0002] During the manufacturing process of industrial products such as machinery or electronics, defects such as pits, wear, scratches, cracks, and yellow spots may occur due to a variety of factors such as processing defects, imperfect manufacturing conditions, and unexpected mechanical failures. These defects increase manufacturing costs, shorten the service life of the finished product, and even cause substantial harm to users. For example, defects in key mechanical or electronic components of a car may be potential safety issues. Therefore, detecting product defects is a key task to improve product quality.

[0003] Compared with manual inspection, automatic defect detection technology has obvious advantages. It can not only reduce the workload of inspectors, but also improve the efficiency, stability and performance of the inspection process, while adapting to various working environments that are not suitable for humans. The AOI system based on deep learning can achieve efficient and stable automatic inspection of industrial parts.

[0004] In the existing deep learning-based AOI system, a specific deep learning-based AOI detection model needs to be trained for industrial components in a domain (for example, an industrial component production line). In order to train the detection model to perform AOI detection tasks, a large number of component images in the domain need to be collected and labeled as training data. However, especially for large enterprises with many types of industrial component production lines, for each AOI task of industrial components, a large amount of image data of industrial components of this type must be collected and labeled, which requires a lot of collection and labeling work. In addition, for newly established industrial component production lines, it takes a long time to collect a large number of defective component images, which requires more labeling time. At least based on the above problems, for AOI detection of industrial components with many types, it is necessary to further improve the development, maintenance, and upgrade efficiency of AOI systems based on deep learning. Summary of the invention

[0005] The following summary is provided to introduce selected concepts in a simplified form that are further described in the detailed description that follows. This summary is not intended to highlight key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] In response to the above problems, the present application provides an AOI detection method, device and system based on a visual basic model and a lightweight AOI task detection model.

[0007] According to one aspect of the present application, a method for AOI is provided, comprising: obtaining an image of a component to be inspected in a first domain, the first domain corresponding to a first industrial component production line; generating a first feature representation of the image based on the image of the component to be inspected by a general feature extraction submodel of a detection model, wherein the detection model includes the general feature extraction submodel and a plurality of task detection submodels related to a specific AOI task of the first domain; generating a plurality of detection results related to a specific AOI task of the first domain based on the first feature representation of the image by the plurality of task detection submodels; and determining a state of the component to be inspected based on the plurality of detection results.

[0008] According to one aspect of the present application, a method for training a detection model for AOI is provided, comprising: training a general feature extraction submodel in the detection model based on images of industrial components in multiple domains as a multi-domain AOI training dataset to obtain a trained general feature extraction submodel, wherein the multiple domains correspond to multiple industrial component production lines, respectively; training multiple task detection submodels in the detection model related to a specific AOI task in the first domain based on images of industrial components in a first domain as a first domain AOI training dataset, the multiple task detection submodels including at least a part of an anomaly detection submodel, an image classification submodel, a target detection submodel and a target segmentation submodel, the first domain corresponding to a first industrial component production line of a first type of industrial component, wherein the trained general feature extraction submodel receives an image in the first domain AOI training dataset and outputs a first feature representation of the image, and the multiple task detection submodels predict multiple detection results related to the specific AOI task in the first domain based on the first feature representation of the image.

[0009] According to one embodiment of this aspect, the method also includes: using a mapping model to map at least a portion of the training data of the first domain AOI training data set to a second domain, wherein the second domain corresponds to a second industrial component production line of the first type of industrial component; and adjusting the trained multiple task detection sub-models related to the specific AOI task of the first domain based on at least a portion of the training data mapped to the second domain.

[0010] According to one aspect of the present application, a device for AOI is provided, comprising: an image acquisition module, for acquiring an image of a component to be inspected in a first domain, the first domain corresponding to a first industrial component production line; a detection module, the detection module comprising a general feature extraction submodule, a plurality of task detection submodules related to a specific AOI task of the first domain, and a judgment module, the general feature extraction submodule generating a first feature representation of the image based on the image of the component to be inspected, the plurality of task detection submodules generating a plurality of detection results related to the specific AOI task of the first domain based on the first feature representation of the image, and the judgment module determining a state of the component to be inspected based on the plurality of detection results according to a predefined rule.

[0011] According to one aspect of the present application, a system for AOI is provided, comprising: an image capture device for capturing an image of an inspected component; one or more processors; and one or more memories, wherein the memories store computer executable instructions, and the instructions, when executed by the one or more processors, perform operations according to an embodiment of the present application.

[0012] According to one aspect of the present application, a computer system for AOI is provided, comprising: one or more processors; and one or more memories, wherein the memories store computer executable instructions, and when the instructions are executed by the one or more processors, the operations described in the embodiments of the present application are performed.

[0013] According to one aspect of the present application, a machine-readable storage medium for AOI is provided, which stores executable instructions. When the instructions are executed, one or more processors perform operations according to the embodiments of the present application.

[0014] According to one aspect of the present application, a computer program product for AOI is provided, which includes executable instructions. When the instructions are executed, one or more processors perform the operations described in the embodiments of the present application.

[0015] By using the technical solution of the present application, a visual base model trained based on industrial component images of multiple domains as an unlabeled data set can extract a universal visual representation of industrial component images that is adapted to multiple domains. When the universal visual representation of the image is used in conjunction with a lightweight task detection head of an AOI detection system, the task detection head can be trained based on a small amount of domain-specific labeled training data, thereby greatly improving the generalization and learning efficiency of the AOI detection system for new tasks. In addition, by integrating the visual base model and multiple task detection heads dedicated to multiple domains for training, the visual base model that has been pre-trained with a large number of industrial component images of multiple domains can be efficiently utilized. In addition, by integrating the visual base model and multiple task detection heads dedicated to multiple domains for detection, the AOI detection processing of multiple production lines can be processed in a pipeline manner through the integrated AOI detection model, thereby further improving the utilization of computing resources. In addition, in the case where multiple production lines for the same type of product need to be established in different locations, the AOI inspection model for a production line for this type of product that has been successfully deployed at one location can be fine-tuned through domain adaptation methods, so that the AOI inspection model can meet the indicators in the environment of the new production line without the need to collect and mark defective products on the new production line, thereby achieving fast and efficient deployment of the AOI system on the new production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A further understanding of the nature and advantages of the present application may be achieved by referring to the following drawings, in which similar components or features may have the same reference numerals.

[0017] Figure 1 A block diagram of an AOI system according to one embodiment is shown.

[0018] Figures 2A to 2D 1 and 2 show block diagrams of an AOI apparatus for industrial components according to an embodiment.

[0019] Figure 3 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0020] Figure 4 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0021] Figure 5 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0022] Figure 6 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0023] Figure 7 A schematic diagram of a contrastive learning method for training a visual base model according to one embodiment is shown.

[0024] Figure 8 A schematic diagram of a masked autoencoder (MAE) learning method for training a visual basis model according to one embodiment is shown.

[0025] Fig. 9 A schematic diagram for training a task detection model according to one embodiment is shown.

[0026] Fig.10 A schematic diagram for training a task detection model according to one embodiment is shown.

[0027] Fig.11 A flow chart of a method for performing AOI according to one embodiment is shown.

[0028] Fig.12 A flowchart of a method for training an inspection model for AOI according to one embodiment is shown.

[0029] Fig.13 A block diagram of an apparatus for AOI of industrial components according to one embodiment is shown.

[0030] Fig.14 A block diagram of a processing system for AOI of industrial parts is shown according to one embodiment. DETAILED DESCRIPTION

[0031] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed can be changed without departing from the scope of protection of the present application. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0032] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0033] Figure 1 A block diagram of an AOI system according to one embodiment is shown.

[0034] The AOI system 100 may be a device for inspecting finished or semi-finished products of industrial parts produced. For example, the AOI system 100 may be deployed on a production line to inspect parts during the manufacturing process. The AOI system 100 may be part of a production line, or may be deployed at a location separate from the production line.

[0035] The AOI system 100 may include a front end 110, which may also be referred to as an optical system or a vision system 110. The front end 110 includes an imaging unit 1110 and an object carrier unit 1120. The imaging unit 1110 may capture an image of an object 10 placed on the carrier unit 1120. In one example, the imaging unit 1110 may include a sensing element and an optical focusing system that cooperate to achieve imaging of the object 10.

[0036] Examples of the carrier unit 1120 may be a mechanical platform, a mechanical arm, a conveyor belt, etc., which may grab, hold, and convey the object 10 so that the imaging unit 1110 may capture an image of the object 10. In one example, the imaging unit 1110 and the carrier unit 1120 may cooperate to capture multiple images of different parts of the object 10.

[0037] Although the carrier unit 1120 is shown as part of the front end 110, it should be understood that in some embodiments, the front end 110 may not actually include the carrier unit 1120, but rather the carrier unit is part of the production line. For example, the object 10 may be a semi-finished component placed on the carrier unit 1120 along the production line, and the imaging unit 1110 is deployed at this part of the production line to inspect the semi-finished component during manufacturing.

[0038] The front end 110 may send an image of the object 10 to the back end 120, which may also be referred to as a processing system or computing system 120. The processing system 120 may be implemented in various ways, for example, the processing system 120 may include one or more processors and / or controllers and one or more memories, and the processors and / or controllers may execute software to perform various operations or functions, such as operations or functions according to various aspects of the present application.

[0039] The processing system 120 may receive image data from the imaging unit 1110 and perform various operations by analyzing the image data. Figure 1 In the example of, the processing system 120 may include an AOI detection module 1210, and the AOI detection module 1210 may perform AOI detection processing on the received image of the object 10. For example, the AOI detection module 1210 may classify the image of the object 10, and the result of the classification may indicate whether the object 10 is a qualified product. For another example, the AOI detection module 1210 may segment the image of the object 10, thereby outputting a mask and a category indicating a defective portion in the image. For another example, the AOI detection module 1210 may recognize the image of the object 10, thereby outputting a bounding box and a category indicating a defective portion in the image. For another example, the AOI detection module 1210 may perform anomaly detection processing on the image of the object 10, thereby outputting an area indicating a defective portion in the image. It can be understood that the AOI detection module 1210 may perform some or all of the above tasks.

[0040] It is understood that the AOI detection module 1210 can be implemented in various ways, for example, it can be implemented as a software module or function. It is understood that in addition to the module 1210, the computing system 120 can also include other modules. For example, the control module in the computing system 120 can be configured to control the operation of the imaging unit 1110 and the carrying unit 1120 so as to capture one or more images of the object 10, for example, the imaging unit 1110 and the carrying unit 1120 can cooperate under the control of the control module to capture multiple images of different parts of the object. It is understood that such a control module can also be implemented in the front end 110. It is understood that the computing system 120 can be co-located with the front end 110 in the AOI system 100, for example, in the housing of the AOI system 100, and can also be located at a location away from the front end 110, for example, the computing system 120 can be implemented on a server or cloud, and the image captured by the front end 110 can be transmitted to the computing system 120 via a wired or wireless communication connection.

[0041] Figures 2A to 2D 1 and 2 show block diagrams of an AOI apparatus for industrial components according to an embodiment.

[0042] like Figure 2A As shown, images I1A and I2A may be images of industrial components 10 on a first production line. Since the industrial components of the first production line are of the same type, the images of these industrial components belong to the same domain D1. In this document, the term domain corresponds to an industrial component production line, that is, one domain corresponds to one production line of one type of industrial component. For example, Figure 2A One type of industrial component shown has a production line in city A and city B, respectively. The production line in city A corresponds to the first domain, and the production line in city B corresponds to the second domain. After receiving the image (e.g., image I1A, I2A) of the industrial component 10 from the imaging unit 1110, the AOI detection module 210A specific to the domain D1 can process the images I1A and I2A to extract feature representations of the images and predict outputs 220A related to the AOI task, such as classification results, segmentation results, and / or recognition results of the images. Those skilled in the art will appreciate that any suitable neural network model can be used to implement the above-mentioned classification tasks, segmentation tasks, and recognition tasks.

[0043] Similarly, Figures 2B to 2D In the embodiment, images I1B and I2B may be images of industrial components 10 on the second production line, belonging to domain D2, images I1C and I2C may be images of industrial components 10 on the third production line, belonging to domain D3, and images I1D and I2D may be images of industrial components 10 on the fourth production line, belonging to domain D4. The AOI detection module 210B specific to domain D2 may process images I1B and I2B to extract feature representations of the images and predict outputs 220B related to the AOI task, the AOI detection module 210C specific to domain D3 may process images I1C and I2C to extract feature representations of the images and predict outputs 220C related to the AOI task, and the AOI detection module 210D specific to domain D4 may process images I1D and I2D to extract feature representations of the images and predict outputs 220D related to the AOI task.

[0044] AOI detection modules 210A to 210D are all specific implementations of AOI detection module 1210. Since AOI detection modules 210A to 210D are all domain-specific AOI detection modules, a large amount of training image data needs to be collected and annotated in the corresponding domain during the training phase of AOI detection modules 210A to 210D. Specifically, a large amount of training image data is collected and annotated in domain D1 to train AOI detection module 210A, a large amount of training image data is collected and annotated in domain D2 to train AOI detection module 210B, and so on.

[0045] Figure 3A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0046] like Figure 3 As shown, images I1A, I2A to I1D, I2D can be Figures 2A to 2D The industrial component images corresponding to the multiple domains D1 to D4 are shown. As described above, the images I1A, I2A to I1D, I2D are, for example, images of industrial components from four different production lines. In one embodiment, Figure 1 The AOI inspection module 1210 shown may include Figure 3 The visual basis model 310, task detection models 320A to 320D and judgment modules 340A to 340D are shown.

[0047] The visual base model 310 may be a self-attention mechanism model. For example, the visual base model 310 may use a hierarchical visual transformer (e.g., Swin, CvTVision, CSwin, SAM, DINOv2, etc.) as a backbone encoder to extract feature representations of images. By pre-training the visual base model 310 based on an unlabeled dataset of industrial component images of multiple domains, the trained visual base model 310 can extract a general visual representation of the industrial component image that is adapted to multiple domains. The general visual representation of the image can be adapted to multiple domains corresponding to the training dataset and multiple unknown domains outside the training dataset. Therefore, the AOI detection module 1210 including the visual base model 310 can learn general AOI capabilities across multiple domains.

[0048] Task detection models 320A to 320D are task detection models specific to domains D1 to D4, respectively. Figure 3 As shown, the task detection models 320A to 320D include multiple task detection sub-models 3210A to 3230A, 3210B to 3230B, 3210C to 3230C, and 3210D to 3230D, respectively. It can be understood that Figure 3 Any of the task detection models 320 shown in FIG. 3 may include a greater or lesser number of task detection sub-models. Figures 2A to 2DThe task detection models 210A to 210D specific to domains D1 to D4 shown, and the task detection models 320A to 320D are lightweight task detection models. Taking the task detection model 320A as an example, after the visual base model 310 receives the image I1A and generates the feature representation FM-A of the image I1A, the task detection sub-models 3210A to 3230A in the task detection model 320A perform the AOI detection task based on the feature representation FM-A and output the detection results 3310A to 3330A. For example, the AOI detection task performed by each of the task detection sub-models 3210A to 3230A may be one of a classification task, a segmentation task, a recognition task, and an anomaly detection task, and each of the corresponding detection results 3310A to 3330A may be one of a classification result, a segmentation result, a recognition result, and an anomaly detection result. Similarly, after the visual base model 310 receives the image I1B and generates the feature representation FM-B of the image I1B, the task detection sub-models 3210B to 3230B in the task detection model 320B perform the AOI detection task based on the feature representation FM-B and output the detection results 3310B to 3330B, and the task detection models 320C and 320D also perform the AOI detection task based on the feature representations FM-C and FM-D of the images I1C and I1D generated by the visual base model 310, respectively.

[0049] Since the pre-trained visual basic model 310 has already performed universal feature extraction on the component images across multiple domains, the task detection models 320A to 320D only need to be implemented using lightweight task models. Figures 2A to 2D Compared with the training of the task detection models 210A to 210D shown in the figure, only a small amount of domain-specific annotated training data is needed to train the task detection sub-models 3210A to 3230A, 3210B to 3230B, 3210C to 3230C, and 3210D to 3230D in the task detection models 320A to 320D, thereby greatly improving the learning efficiency of the AOI detection system for new tasks. In addition, by integrating the visual base model and multiple task detection heads dedicated to multiple domains for training, the visual base model that has been pre-trained with a large number of industrial component images in multiple domains can be efficiently utilized, unlike the prior art that requires training a specific deep learning-based AOI detection model for each domain.

[0050] The judgment modules 340A to 340B respectively judge the state of the detected component according to predefined rules based on the detection results 3310A to 3330A, 3310B to 3330B, 3310C to 3330C, 3310D to 3330D output by the corresponding task detection sub-models 3210A to 3230A, 3210B to 3230B, 3210C to 3230C, 3210D to 3230D. Taking the AOI detection of the component of the first domain D1 as an example, for example, the task detection sub-models 3210A to 3230A can respectively detect the regions of various types of defects of the component based on the image I1A of the detected component. The judgment rules corresponding to each type of defect can be predefined in the judgment model 340A. For example, exemplary judgment rules are shown in Table 1 below. In this example, the task detection sub-models 3210A to 3230A output the regions of the first type to the third type of defects respectively based on the image of the inspected component. For example, the first type to the third type of defects are yellow spots, holes and cracks, respectively. In the example shown in Table 1, corresponding position thresholds, size thresholds and area thresholds can be defined, and the size threshold includes a length threshold and a width threshold. In one embodiment, the position threshold represents a specific position area of ​​the inspected component, wherein if the corresponding type of defect appears in the specific position area represented by the position threshold, the inspected component is judged to be an unqualified component. In one embodiment, the length threshold and the width threshold represent that if the length and / or width of the corresponding type of defect area is greater than the length threshold and / or the width threshold, the inspected component is judged to be an unqualified component. In one embodiment, the area threshold represents that if the area of ​​the corresponding type of defect area is greater than the area threshold, the inspected component is judged to be an unqualified component. It can be understood that, although the position threshold, length threshold, width threshold and area threshold are given as examples for each defect type in the example of Table 1, in a specific implementation, the judgment rule for a specific defect type may include only a part of the position threshold, length threshold, width threshold and area threshold, or may include other judgment conditions. It can be understood that the task detection sub-models 3210A to 3230A can detect more types of potential defect areas, such as welds, solder joints, solder spatter spots, ripples, etc., and corresponding judgment rules can be set for each type of potential defect area to determine whether the detected component is a substandard product. Table 1

[0051] exist Figure 3In the embodiment of the AOI system shown, by integrating a visual base model and multiple task detection heads dedicated to multiple domains to perform AOI detection tasks for multiple production lines, the AOI detection processing of multiple production lines can be processed in a pipeline manner through the integrated AOI detection model, thereby potentially further improving the utilization rate of the computing resources of the computing device 120.

[0052] Those skilled in the art will appreciate that although Figure 3 In the illustrated embodiment, the AOI detection module 1210 includes a visual base model 310 and a plurality of domain-specific task detection models 320A to 320D, but in other embodiments, the AOI detection module 1210 may also include a visual base model 310 and a task detection model 320, which is used to process AOI detection of industrial parts from a domain (e.g., a production line).

[0053] Figure 4 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown. Figure 4 Zhongyu Figures 1 to 3 The same or similar reference numerals shown represent the same or similar elements. Figure 1 The AOI inspection module 1210 shown may include Figure 4 Shown are a visual basic model 310, task detection sub-models 3210 to 3230, an image cropping module 350, and judgment modules 340A to 340D.

[0054] The type of component to be inspected is a valve assembly solenoid, which is generally a disc shape, such as Figure 4 As shown, image I represents three pictures of a component taken from three angles, so as to cover all parts of the component that need to be inspected. The production line to which the component belongs corresponds to a domain. It can be understood that the three pictures shown can be processed in parallel or in series. In the description of this article, the processing of one of the pictures is taken as an example for explanation.

[0055] The visual basis model 310 processes the image I to generate a feature representation FM-I of the image I. The segmentation model 3210 generates a segmentation result 3310 related to the potential defect based on the feature representation FM-I. Figure 4In the example shown, the segmentation result 3310 includes a mask corresponding to a weld (denoted as SM, Seam in the figure) and a mask corresponding to a solder spot (denoted as ST, Spot in the figure), which indicates the segmented area relative to the background portion (denoted as BR, Background in the figure). In this example, the segmentation model 3210 is configured to detect two types of potential defects, welds and solder spots. It can be understood that the segmentation model 3210 can also be configured to detect only welds, because the inspection of welds is crucial in the AOI inspection of this type of components. It can be understood that welds and solder spots are only potential defect areas, which does not mean that they must be defects that cause the component to be detected as a defective product. Therefore, in the judgment component 340, it is further judged whether it causes the component to be unqualified based on its position, size and / or area.

[0056] The image clipping module 350 clips the image I based on the segmentation result 3310 to obtain a partial image PI, which can be called an interesting part. For this particular component, the weld and the welding spot are the most important factors for judging whether the product is qualified, so the image clipping module 350 clips the partial image PI containing the weld and the welding spot from the image I based on the segmentation result 3310.

[0057] The visual base model 310 processes the partial image PI to generate a feature representation FM-PI of the partial image PI. The anomaly detection model 3220 generates an anomaly detection result 3320 related to potential defects based on the feature representation FM-PI. The anomaly detection model 3220 can be implemented by an out-of-distribution model (OOD), which detects abnormal parts in the image PI by detecting sample points that are inconsistent with the sample points of the normal distribution. Figure 4 In the example shown, the anomaly detection model 3220 is configured to detect two types of potential defects, holes and ripples. It is understood that the segmentation model 3210 can also be configured to detect fewer or more types of defects. It is understood that holes and ripples are only potential defect areas, which does not mean that they must be defects that cause the component to be detected as a defective product. Therefore, in the judgment component 340, it is further determined whether it causes the component to be unqualified based on its position, size and / or area.

[0058] The segmentation model 3230 generates segmented regions 3330 associated with potential defects based on the feature representation FM-PI. Figure 4In the example shown, the segmentation result 3330 includes a mask corresponding to a crack (indicated as CK, Crack in the figure). In this example, the segmentation model 3230 is configured to detect potential defects of the type of crack. It is understood that the segmentation model 3210 can also be configured to detect more types of defects, for example, the segmentation model 3210 can also be configured to detect holes. It is understood that cracks and holes are only potential defect areas, which does not mean that they must be defects that cause the component to be detected as a defective product. Therefore, in the judgment component 340, it is further judged whether it causes the component to be unqualified based on its position, size and / or area.

[0059] The judgment module 340 judges whether the inspected component is a defective product according to the inspection results 3310, 3320 and 3330. For example, the judgment module 340 can judge whether the component is a defective product according to the position, size and / or area of ​​various types of potential defect areas according to the pre-configured judgment rules as described in Table 1 above.

[0060] Figure 5 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0061] Figure 5 Zhongyu Figures 1 to 4 The same or similar reference numerals shown in the figure represent the same or similar units, and the corresponding parts will not be described in detail. Figure 4 Compared to the embodiment shown, Figure 5 The AOI inspection module 1210 in the AOI system of the embodiment further includes a segmentation model 3240. The segmentation model 3240 generates a segmentation region 3340 related to a potential defect based on the feature representation FM-I. Figure 5 In the example shown, the segmentation result 3340 includes a mask corresponding to a yellow spot (denoted as YS in the figure). In this example, the segmentation model 3240 is configured to detect potential defects of the type of yellow spot. It is understood that the segmentation model 3240 can also be configured to detect more types of defects, for example, the segmentation model 3240 can also be configured to detect solder spatter spots. It is understood that yellow spots and spatter spots are only potential defect areas, which does not mean that they must be defects that cause the component to be detected as a defective product. Therefore, in the judgment component 340, it is further judged whether it causes the component to be unqualified based on its position, size and / or area.

[0062] The judgment module 340 judges whether the inspected component is a defective product according to the inspection results 3310, 3320, 3330 and 3240. For example, the judgment module 340 can judge whether the component is a defective product according to the position, size and / or area of ​​various types of potential defect areas according to the pre-configured judgment rules as described in Table 1 above.

[0063] Figure 6 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0064] Figure 6 Zhongyu Figures 1 to 5 The same or similar reference numerals shown in the figure represent the same or similar units, and the corresponding parts will not be described in detail. Figure 5 Compared to the embodiment shown, Figure 6 The AOI detection module 1210 in the AOI system also includes a segmentation model 3250, and the segmentation model 3230 and the segmentation model 3250 together constitute a combined segmentation model 3235. The segmentation model 3230 and the segmentation model 3250 are implemented by different types of neural network models. For example, the segmentation model 3230 can be implemented by a DeepLab model, and the segmentation model 3250 can be implemented by a feature pyramid pooling (FPN) model. In this example, the segmentation model 3230 and the segmentation model 3250 are respectively trained to detect cracks, and the output segmentation result 3330 is obtained by combining the output results of the two (for example, by weighted averaging the output results of the two). By adopting this combined segmentation model, the robustness of the segmentation result can be improved, while reducing the problem of overconfidence that may occur in a single segmentation model. Therefore, for important defects such as cracks, the accuracy and robustness of detection can be further improved. It is understandable that the segmentation model 3235 can also be configured to detect more defect types. For example, the segmentation model 3235 can also be configured to detect holes.

[0065] The judgment module 340 judges whether the inspected component is a defective product according to the inspection results 3310 , 3320 , 3330 and 3240 .

[0066] exist Figures 4 to 6 In the various embodiments shown, for example, the segmentation modules 3210 and 3240 can be implemented by the FPN model and the DeepLab model, respectively, the segmentation modules 3230 and 3250 can be implemented by the FPN model and the DeepLab model, respectively, and the anomaly detection module 3220 can be implemented by the OOD model. By arranging the anomaly detection module 3220 in the task-specific detection module 320, the escape rate of detection can be improved. In order to train a task-specific inspection module for a domain, it is necessary to collect pictures of defective parts in the pictures of industrial parts corresponding to the domain and perform corresponding annotations to obtain labeled training data. The collection and annotation of the training data is usually time-consuming, and it is possible that the collected and annotated training data set cannot cover all defect types. The training of the OOD model can be based on an unlabeled data set, so in Figures 4 to 6 In the illustrated embodiment, by utilizing this feature of the OOD model, it is possible to help detect more types of potential defects, thereby improving the escape rate of detection and reducing the probability that actually unqualified products are not detected. It can be understood that Figures 4 to 6 The various models shown are not limited to being implemented using the neural network model exemplified above, but may also be implemented using other suitable models.

[0067] Figure 7 A schematic diagram of a contrastive learning method for training a visual base model according to one embodiment is shown.

[0068] In order to train the visual base model 310, it is necessary to establish a large-scale dataset of industrial component images including multiple domains. Figure 3 In the example of the AOI system shown, for a large enterprise with many kinds of industrial component production lines, the imaging unit 1110 deployed on each production line can obtain images of multiple kinds of industrial components as training data sets for pre-training the visual base model. For example, in one embodiment, millions of industrial component images can be collected from multiple production lines as AOI training data sets for pre-training the visual base model.

[0069] In one embodiment, it is possible to use Figure 7 The contrastive learning method shown is used to train the visual base model 310 to learn to extract feature representations of industrial component images of multiple domains. Figure 7 Taking the industrial component image I1 as an example, the data enhancement module 710 performs enhancement processing on the image I1 to obtain an image pair (I1-1, I1-2). For example, the data enhancement module 710 can obtain the image pair (I1-1, I1-2) by performing color change, brightness change, random shearing, flipping, rotation, etc. on the image I1, and the image pair can be called a positive pair.

[0070] The visual basis model 310 encodes the images I1-1 and I1-2 respectively to obtain the feature representations FM-1 and FM-2 of the corresponding images. The comparison module 720 obtains the similarity between the two images I1-1 and I1-2 based on the feature representations FM-1 and FM-2. For example, the similarity can be the cosine similarity between the feature vectors corresponding to the two images I1-1 and I1-2. In one example, the comparison module 720 can calculate the cosine similarity between the feature representations FM-1 and FM-2 as the similarity S1 between the two images I1-1 and I1-2. In another example, the comparison module 720 can also include an additional neural network layer for converting FM-1 and FM-2 to feature representations FM-1Z and FM-2Z in another space, and the comparison module 720 calculates the cosine similarity between the feature representations FM-1Z and FM-2Z as the similarity S1 between the two images I1-1 and I1-2.

[0071] The loss determination module 730 can determine the loss value L1 based on the similarity S1 between the two images I1-1 and I1-2. For example, the loss value L1 can be determined based on the similarity of the positive pair (I1-1, I1-2) and the similarity of the enhanced image and the other images in a batch constituting a negative pair. Determining the loss value based on the similarity of the positive pair and the negative pair is a commonly used technical means in the field of image processing, so it will not be described in detail. The visual base model 310 can then be updated based on the comparative learning loss value L1. For example, the known AdamW optimizer can be used to update the visual base model 310 based on the loss L1.

[0072] Figure 8 A schematic diagram of a masked autoencoder (MAE) learning method for training a visual basis model according to one embodiment is shown.

[0073] In one embodiment, it is possible to use Figure 8 The MAE method shown is used to train the visual base model 310 to learn to extract feature representations of industrial component images in multiple domains. Figure 8 Taking the industrial component image I1 as an example, the masking module 810 performs masking processing on the image I1 to obtain a masked image I1-M. For example, the masking module 810 divides the image I1 into small blocks, and then randomly selects a portion of the small blocks to retain, and masks all the remaining small blocks to obtain the masked image I1-M.

[0074] The visual basis model 310 encodes the masked image I1-M to obtain the feature representation FM-M of the masked image. The decoder 820 decodes the restored image I1-R based on the feature representation FM-M. The loss determination module 830 determines the MAE loss value L2 based on the restored image I1-R and the original image I1. For example, the loss determination module 830 can use the mean square error (MSE) function as the loss function to determine the MAE loss value L2, that is, the original pixels in the original image I1 and the reconstructed pixels in the restored image I1-R are subtracted and then the square sum is calculated. Then, the visual basis model 310 can be updated based on the MAE loss value L2. For example, the known AdamW optimizer can be used to update the visual basis model 310 based on the loss L2.

[0075] In one embodiment, both Figure 7 and 8 The contrastive learning method and MAE method shown are used to train the visual base model 310. For example, the weighted sum of the loss values ​​L1 and L2 can be used as the loss value to update the basic visual module.

[0076] Those skilled in the art will appreciate that any appropriate training method may be used to train the visual base model 310. The visual base model 310, which is trained with a large-scale dataset of industrial component images of multiple domains, has learned both domain-specific feature representations and general feature representations covering various domains. Therefore, the feature representations of industrial component images extracted by the visual base model 310 as a feature extractor can help the downstream domain-specific AOI detection modules 320 to learn AOI detection tasks adapted to various domains through a lightweight domain-specific adaptive training process.

[0077] Fig. 9 A schematic diagram for training a task detection model 320 according to one embodiment is shown.

[0078] by Figures 3 to 6 The domain D1-specific task detection submodel 3210 is used as an example to illustrate Fig. 9 The task detection sub-model 3210 is Figure 4-6 The embodiment shown is a segmentation model. It can be understood that in other embodiments, Fig. 9The training process shown can also be used to train other types of detection models, such as classification models, target recognition models, etc. In one embodiment, the classification model can be a multi-layer perceptron (MLP) model, the segmentation model can be a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism, and the target recognition model can be a bounding box decoder with classification results of a CNN model or a decoder based on an attention mechanism. Each task detection sub-model can be trained based on a set of images collected and annotated in domain D1 as a training data set. In one embodiment, a lightweight task detection sub-model can be trained based on a small amount of training data sets specific to domain D1. For example, the small amount of training data sets can be a data set in units of hundreds of images.

[0079] The visual base model 310 encodes the image I1 to obtain a feature representation FM of the image. The task detection model 3210 obtains a prediction P about the detection task based on the feature representation. For example, the segmentation model 3210 obtains a prediction 3310 about the detection task based on the feature representation. The loss determination module 930 determines a loss value L3 based on the prediction result P1 and the corresponding label representing the annotated detection result. The loss value L3 can be determined by various appropriate methods for determining the supervised loss using the prediction value and the label value. Then, the task detection submodel 3210 can be updated based on the supervised loss value L3. For example, the known AdamW optimizer can be used to update the task detection submodel 3210 based on the loss L3. In one embodiment, the parameters of the pre-trained visual base model 310 are frozen during the process of updating the task detection model 3210. In another embodiment, a small number of parameters of the pre-trained visual base model 310 can be updated simultaneously during the process of updating the task detection model 3210.

[0080] Fig.10A schematic diagram of a task detection model 320 for training according to an embodiment is shown. For large industrial parts manufacturing companies, it may be necessary to establish multiple production lines in different locations for the same type of product. One problem that arises from this is that, if the AOI solution for a production line of this type of product has been successfully deployed at one location, if the AOI solution is to be transferred to a new production line, the environments of the two production lines are almost impossible to be exactly the same. For example, there may be differences in the equipment used for imaging, ambient light, etc., so the original AOI model may not meet the indicators required by the new production line, and therefore the AOI detection model needs to be retrained. Retraining of the AOI detection model requires collecting images of defective products on the new production line and annotating the collected images to obtain training data. However, the collection and annotation of images of defective products takes a long time. In order to solve this problem, in this embodiment, the training data corresponding to the first domain of the original production line of the same type is mapped to the second domain corresponding to the new production line by adopting a domain adaptation method, and then using Fig. 9 The training process shown further trains the task detection model 320. Since the first domain and the second domain correspond to two production lines of the same type of product, the further training process is a fine-tuning of the task detection model 320 that has been trained in the first domain, and the task detection model obtained by the fine-tuning or further training has good performance for AOI detection of components of the new production line.

[0081] Can be used Fig.10 The mapping model G shown S-T 1010 maps the training data of the first domain corresponding to the original production line of the same type to the second domain corresponding to the new production line. Fig.10 The mapping model G is obtained according to one embodiment. S-T 1010 is a schematic process. The mapping model G can be trained based on unlabeled images of the first domain (source domain) and unlabeled images of the second domain (target domain). S-T 1010. In one embodiment, the mapping model G S-T 1010 may include a general feature extraction submodel 310 and a decoder model, wherein the general feature extraction submodel 310 receives an image SI 1 of a source domain and generates a corresponding feature representation FSI 1, and the decoder model generates an image MSI 2 mapped to a target domain based on the feature representation. Another mapping model G T-S 1020 with Mapping Model G S-T1010 has a similar structure, which may include a general feature extraction submodel 310 and a decoder model, wherein the general feature extraction submodel 310 receives the image MSI 2 and generates a corresponding feature representation FMSI 2, and the decoder model generates a reconstructed image RSI 3 mapped to the source domain based on the feature representation. Fig.10 In the illustrated embodiment, a circulation loss value CL may be generated based on the source image SI 1 and the reconstructed source image RSI 3 , and a semantic consistency loss SCL may be generated based on the feature representation FSI 1 and the feature representation FMSI 2 .

[0082] Discriminator D T 1030 is used to determine whether the image MSI 2 mapped to the target domain and the image TI 4 collected in the target domain are from the same domain, and to generate a pixel-level Generative Adversarial Network (GAN) loss GAN L1 based on the determination result. The general feature extraction submodel 310 generates a feature representation FTI 4 of the target domain image TI 4, and the discriminator D T 1040 is used to determine whether the feature representation FMSI 2 of the image MSI 2 mapped to the target domain and the feature representation FTI 4 of the image TI 4 collected in the target domain are from the same domain, and generate a feature-level loss GAN L2 based on the determination result. The mapping model G is updated based on the above-mentioned cycle loss CL, semantic consistency loss SCL, GAN L1 and GAN L2. S-T Decoder model and mapping model G in 1010 T-S Decoder model and discriminator D in 1020 T 1030 and, discriminator D T 1040, thus finally obtaining the mapping model G S-T 1010.

[0083] Since the unlabeled target domain image TI 4 is easy to collect, the target domain image can be obtained in a very short time at the beginning of the trial operation of the new production line to obtain the mapping model G S-T 1010, and then through the mapping model G S-T 1010 maps the labeled training data of the original production line to the target domain corresponding to the new production line, thereby based on the mapped training data through, for example Fig. 9 The training process shown completes fine-tuning or further training of the task detection model 320.

[0084] Understandably, Fig.10 The process shown is used to obtain the mapping model G S-T A specific example of 1010, the mapping model G can also be obtained by other appropriate methodsS-T 1010.

[0085] Fig.11 A flow chart of a method for performing AOI according to one embodiment is shown.

[0086] In step 1110, an image of an inspected component of a first domain is obtained, the first domain corresponding to a first industrial component production line.

[0087] In step 1120, a first feature representation of the image is generated based on the image of the inspected component by a general feature extraction submodel of the inspection model, wherein the inspection model includes the general feature extraction submodel and a plurality of task inspection submodels related to the specific AOI tasks of the first domain.

[0088] At step 1130 , the multiple task detection sub-models generate multiple detection results related to the specific AOI task of the first domain based on the first feature representation of the image.

[0089] In step 1140, the state of the detected component is determined based on the multiple detection results according to predefined rules.

[0090] In one embodiment, at step 1140 , it is determined whether the inspected component is a defective component based on the position, size and / or area of ​​the defective region indicated by at least some of the inspection results.

[0091] In one embodiment, the multiple task detection submodels include at least a portion of an anomaly detection submodel, an image classification submodel, a target detection submodel, and a target segmentation submodel. In one embodiment, the multiple task detection submodels include an anomaly detection submodel and at least a portion of an image classification submodel, a target detection submodel, and a target segmentation submodel.

[0092] In one embodiment, the multiple task detection sub-models include a first segmentation sub-model, an anomaly detection sub-model, and a second segmentation sub-model. In step 1130, the first segmentation sub-model segments a region of at least a first type of defect based on the image of the inspected component; a partial image is cut out from the image of the inspected component based on the segmentation result of the first segmentation sub-model; the anomaly detection sub-model detects a region of at least a second type of defect based on the partial image of the inspected component; and the second segmentation sub-model segments a region of at least a third type of defect based on the partial image of the inspected component. In step 1140, it is determined whether the inspected component is a defective component based on the region of at least the first type of defect, the region of at least the second type of defect, and the region of at least the third type of defect.

[0093] In one embodiment, the plurality of task detection sub-models further include a third segmentation sub-model and a fourth segmentation sub-model, wherein the fourth segmentation sub-model and the second segmentation sub-model constitute a combined segmentation sub-model. In step 1130, the third segmentation sub-model segments a region of at least a fourth type of defect based on the image of the inspected component; and the combined segmentation sub-model segments a region of at least the third type of defect based on the partial image of the inspected component. In step 1140, it is determined whether the inspected component is a defective component based on the region of at least the fourth type of defect.

[0094] In one embodiment, the region of at least the third type of defect segmented out by the combined segmentation sub-model based on the partial image of the inspected component includes: segmenting out the region of at least the third type of defect by the second segmentation sub-model based on the partial image of the inspected component; segmenting out the region of at least the third type of defect by the fourth segmentation sub-model based on the partial image of the inspected component; and synthesizing, by the combination module, the region of at least the third type of defect segmented out by the second segmentation sub-model and the region of at least the third type of defect segmented out by the fourth segmentation sub-model to obtain the region of at least the third type of defect segmented out by the combination module.

[0095] In one embodiment, the area of ​​at least the first type of defects includes at least one of a weld and a weld point, the area of ​​at least the second type of defects includes at least one of a hole and a corrugation, the area of ​​at least the third type of defects includes at least one of a hole and a crack, and the area of ​​at least the fourth type of defects includes at least one of a yellow spot and a solder spatter spot.

[0096] Fig.12 A flowchart of a method for training an inspection model for AOI according to one embodiment is shown.

[0097] In step 1210, a general feature extraction sub-model in the detection model is trained based on images of industrial components in multiple domains as a multi-domain AOI training dataset to obtain a trained general feature extraction sub-model, wherein the multiple domains correspond to multiple industrial component production lines respectively.

[0098] In step 1220, multiple task detection sub-models related to the specific AOI task of the first domain in the detection model are trained based on images of industrial components in the first domain as the first domain AOI training data set, and the multiple task detection sub-models include at least a part of anomaly detection sub-model, image classification sub-model, target detection sub-model and target segmentation sub-model, and the first domain corresponds to a first industrial component production line of a first type of industrial component, wherein the trained general feature extraction sub-model receives the image in the first domain AOI training data set and outputs a first feature representation of the image, and the multiple task detection sub-models predict multiple detection results related to the specific AOI task of the first domain based on the first feature representation of the image.

[0099] In one embodiment, method 1200 further includes step 1230. In step 1230, at least a portion of the training data of the first domain AOI training data set is mapped to a second domain using a mapping model, wherein the second domain corresponds to a second industrial component production line of the first type of industrial component, and the trained multiple task detection sub-models related to the specific AOI task of the first domain are adjusted based on the at least a portion of the training data mapped to the second domain. For example, the multiple task detection sub-models related to the specific AOI task of the first domain are adjusted or further trained based on the at least a portion of the training data mapped to the second domain through the operation described in step 1220.

[0100] In one embodiment, the method 1200 further includes: training the mapping model based on a first image set of the first type of industrial component in the first domain and a second image set of the first type of industrial component in the second domain. In one embodiment, the mapping model includes the trained general feature extraction submodel and a decoder model, the trained general feature extraction submodel generates a first feature representation of the image of the first type of industrial component in the first domain, and the decoder model generates a corresponding image mapped to the second domain based on the first feature representation.

[0101] In one embodiment, in step 1220, the plurality of task detection sub-models are trained based on the images and corresponding labels in the first domain AOI training dataset. In step 1230, the images in the at least a portion of the training data in the first domain AOI training dataset are mapped to the second domain using the mapping model, wherein the at least a portion of the training data mapped to the second domain includes the images in the at least a portion of the training data mapped to the second domain and the corresponding labels.

[0102] In one embodiment, in step 1210, data enhancement is performed on the unlabeled multi-domain AOI training dataset to obtain a set of multi-domain AOI training image pairs, and the general feature extraction sub-model is trained using a contrastive learning method based on the multi-domain AOI training image pair set; and / or the general feature extraction sub-model is trained using a masked autoencoder (MAE) method based on the unlabeled multi-domain AOI training dataset.

[0103] In one embodiment, in step 1210, training the general feature extraction submodel based on the multi-domain AOI training image pair set using a contrastive learning method includes: receiving an image pair in the multi-domain AOI training image pair set by the general feature extraction submodel and outputting a first feature representation of the image pair, obtaining a similarity of the image pair based on the first feature representation of the image pair by a contrastive learning module, obtaining a contrastive learning loss based on the similarity of the image pair, and updating the general feature extraction submodel based on the contrastive learning loss. In one embodiment, training the general feature extraction submodel using a MAE method in step 1210 includes: masking an original image in the multi-domain AOI training dataset to obtain a masked image, receiving the masked image by the general feature extraction submodel and outputting a first feature representation of the masked image, predicting a restored image based on the first feature representation of the masked image by a decoder module, obtaining a MAE loss based on the restored image and the original image, and updating the general feature extraction submodel based on the MAE loss.

[0104] Fig.13 A block diagram of an apparatus for AOI of industrial components according to an embodiment is shown. The apparatus 1300 includes: an image acquisition module 1310, for acquiring an image of an inspected component of a first domain, the first domain corresponding to a first industrial component production line; a detection module 1320, the detection module including a general feature extraction submodule, a plurality of task detection submodules related to a specific AOI task of the first domain, and a judgment module, the general feature extraction submodule generating a first feature representation of the image based on the image of the inspected component, the plurality of task detection submodules generating a plurality of detection results related to a specific AOI task of the first domain based on the first feature representation of the image, and the judgment module determining the state of the inspected component based on the plurality of detection results according to a predefined rule.

[0105] In one embodiment, the multiple task detection submodules include a first segmentation submodule, an anomaly detection submodule, and a second segmentation submodule. The first segmentation submodule segments an area of ​​at least a first type of defect based on the image of the detected component; the anomaly detection submodule detects an area of ​​at least a second type of defect based on a partial image cut out from the image of the detected component; the second segmentation submodule segments an area of ​​at least a third type of defect based on the partial image of the detected component; and the judgment module determines whether the detected component is a defective component based on the area of ​​at least the first type of defect, the area of ​​at least the second type of defect, and the area of ​​at least the third type of defect.

[0106] In one embodiment, the plurality of task detection submodules further include a third segmentation submodule and a fourth segmentation submodule. The fourth segmentation submodule and the second segmentation submodule together constitute a combined segmentation submodule. The third segmentation submodule segments an area of ​​at least a fourth type of defect based on the image of the detected component; the combined segmentation submodule segments an area of ​​at least the third type of defect based on the partial image of the detected component; the judgment module also determines whether the detected component is a defective component based on the area of ​​at least the fourth type of defect.

[0107] In one embodiment, Fig.13 The detection module 1320 shown can be a combination of the above Figure 1 , 3 The detection model 1210 described in 6, wherein the general feature extraction submodule and the task detection submodule can be respectively Figures 3 to 6 Described visual basis model 310 and task detection model 320.

[0108] Fig.14 14 shows a block diagram of a processing system for AOI of industrial components according to one embodiment. The control system or processing system 1400 may include one or more control units or processing units 1410 that execute one or more machine-readable instructions stored or encoded in a machine-readable storage medium (i.e., memory 1420). Although not described herein, Fig.14 However, those skilled in the art will appreciate that the control system 1400 may include various other components, such as various communication modules, bus modules, and possible user interface modules. In one embodiment, the processing unit 1410 is configured to execute the above combined Figure 1-13 Describes the various operations and functions.

[0109] According to one embodiment, a program product such as a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium may have instructions, which, when executed by the processing unit 1410, can execute the above combined with the various embodiments of the present application. Figure 1-13 Describes the various operations and functions.

[0110] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "example" used throughout this specification may mean "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0111] The above description of the present application is provided to enable any person of ordinary skill in the art to implement or use the present application. It is obvious to a person of ordinary skill in the art that various modifications are made to the present application, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present application. Therefore, the present application is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for automatic optical inspection (AOI), comprising: obtaining an image of an inspected component of a first domain, the first domain corresponding to a first industrial component production line; Generate a first feature representation of the image based on the image of the inspected component by a general feature extraction sub-model of an inspection model, wherein the inspection model includes the general feature extraction sub-model and a plurality of task inspection sub-models related to specific AOI tasks of the first domain; generating, by the plurality of task detection sub-models, a plurality of detection results related to a specific AOI task of the first domain based on the first feature representation of the image; A state of the detected component is determined based on the plurality of detection results.

2. The method of claim 1, wherein determining the state of the detected component based on the plurality of detection results further comprises: Whether the inspected component is a defective component is determined based on the position, size and / or area of ​​the defective region indicated by at least some of the multiple inspection results.

3. The method of claim 2, wherein: The multiple task detection sub-models include at least a part of an anomaly detection sub-model, an image classification sub-model, a target detection sub-model and a target segmentation sub-model.

4. The method of claim 2, wherein: The multiple task detection sub-models include a first segmentation sub-model, an anomaly detection sub-model, and a second segmentation sub-model, wherein the multiple task detection sub-models generate multiple detection results related to the specific AOI task of the first domain based on the first feature representation of the image, including: Segmenting at least a first type of defect region based on the image of the inspected component by the first segmentation sub-model; Cutting out a partial image from the image of the detected component based on the segmentation result of the first segmentation sub-model; Detecting, by the anomaly detection sub-model, an area of ​​at least a second type of defect based on the partial image of the inspected component; Segmenting at least a third type of defect region based on the partial image of the inspected component by the second segmentation sub-model; Wherein, determining the state of the inspected component based on the multiple inspection results includes: determining whether the inspected component is a defective component based on the area of ​​the at least first type of defect, the area of ​​the at least second type of defect and the area of ​​the at least third type of defect.

5. The method of claim 4, wherein: The multiple task detection sub-models further include a third segmentation sub-model and a fourth segmentation sub-model, wherein the fourth segmentation sub-model and the second segmentation sub-model constitute a combined segmentation sub-model, wherein the multiple task detection sub-models generate multiple detection results related to the specific AOI task of the first domain based on the first feature representation of the image, and further include: Segmenting a region of at least a fourth type of defect based on the image of the inspected component by the third segmentation sub-model; Segmenting the region of the at least third type of defect based on the partial image of the inspected component by the combined segmentation sub-model; Wherein, determining the state of the inspected component based on the multiple inspection results further includes: determining whether the inspected component is a defective component based on the area of ​​the at least fourth type of defect.

6. The method of claim 5, wherein: The step of segmenting the area of ​​at least the third type of defect based on the partial image of the inspected component by the combined segmentation sub-model comprises: Segmenting at least a third type of defect region based on the partial image of the inspected component by the second segmentation sub-model; Segmenting at least a third type of defect region based on the partial image of the inspected component by the fourth segmentation sub-model; The region of at least the third type of defect segmented by the second segmentation sub-model and the region of at least the third type of defect segmented by the fourth segmentation sub-model are synthesized to obtain the region of at least the third type of defect segmented by the combined segmentation sub-model.

7. The method according to claim 4 or 5, wherein: The area of ​​at least the first type of defects includes at least one of a weld and a weld point, the area of ​​at least the second type of defects includes at least one of a hole and a corrugation, the area of ​​at least the third type of defects includes at least one of a hole and a crack, and the area of ​​at least the fourth type of defects includes at least one of a yellow spot and a solder spatter spot.

8. A method for training an inspection model for automatic optical inspection (AOI), comprising: Training the general feature extraction sub-model in the detection model based on images of industrial components in multiple domains as multi-domain AOI training data sets to obtain a trained general feature extraction sub-model, wherein the multiple domains correspond to multiple industrial component production lines respectively; Based on images of industrial components in a first domain as a first domain AOI training data set, multiple task detection sub-models related to specific AOI tasks in the detection model are trained, the multiple task detection sub-models including at least a part of an anomaly detection sub-model, an image classification sub-model, a target detection sub-model and a target segmentation sub-model, and the first domain corresponds to a first industrial component production line of a first type of industrial components, wherein the trained general feature extraction sub-model receives images in the first domain AOI training data set and outputs a first feature representation of the image, and the multiple task detection sub-models predict multiple detection results related to the specific AOI tasks of the first domain based on the first feature representation of the image.

9. The method of claim 8, further comprising: Mapping at least a portion of the training data of the first domain AOI training data set to a second domain using a mapping model, wherein the second domain corresponds to a second industrial component production line of the first type of industrial component; The trained plurality of task detection sub-models related to a specific AOI task in the first domain are adjusted based on at least a portion of the training data mapped to the second domain.

10. The method of claim 9, further comprising: The mapping model is trained based on a first set of images of the first type of industrial components in the first domain and a second set of images of the first type of industrial components in the second domain.

11. The method according to claim 9 or 10, wherein: The mapping model includes the trained general feature extraction sub-model and a decoder model, wherein the trained general feature extraction sub-model generates a first feature representation of an image of the first type of industrial component in the first domain, and the decoder model generates a corresponding image mapped to the second domain based on the first feature representation.

12. The method of claim 11, wherein: The step of training the plurality of task detection sub-models in the detection model related to the specific AOI task of the first domain based on the images of the industrial components in the first domain as the first domain AOI training data set comprises: training the plurality of task detection sub-models based on the images and corresponding labels in the first domain AOI training data set, Wherein, mapping at least a portion of the training data of the first domain AOI training data set to the second domain using the mapping model includes: mapping images in the at least a portion of the training data of the first domain AOI training data set to the second domain using the mapping model, wherein the at least a portion of the training data mapped to the second domain includes the images in the at least a portion of the training data mapped to the second domain and the corresponding labels.

13. The method of claim 8, wherein: The training of the general feature extraction sub-model in the detection model based on the multi-domain AOI training data set includes: Performing data enhancement on the unlabeled multi-domain AOI training dataset to obtain a multi-domain AOI training image pair set, and training the general feature extraction sub-model using a contrastive learning method based on the multi-domain AOI training image pair set; and / or Based on the unlabeled multi-domain AOI training dataset, the general feature extraction sub-model is trained using a masked autoencoder (MAE) method.

14. An apparatus for automatic optical inspection (AOI), comprising: An image acquisition module, configured to acquire an image of an inspected component of a first domain, the first domain corresponding to a first industrial component production line; A detection module, the detection module includes a general feature extraction submodule, multiple task detection submodules related to the specific AOI task of the first domain, and a judgment module, the general feature extraction submodule generates a first feature representation of the image based on the image of the detected component, the multiple task detection submodules generate multiple detection results related to the specific AOI task of the first domain based on the first feature representation of the image, and the judgment module determines the state of the detected component based on the multiple detection results according to predefined rules.

15. The device of claim 14, wherein: The multiple task detection submodules include a first segmentation submodule, an anomaly detection submodule, and a second segmentation submodule, The first segmentation submodule segments a region of at least a first type of defect based on the image of the inspected component; The abnormality detection submodule detects an area of ​​at least a second type of defect based on a partial image cut out from the image of the inspected component; The second segmentation submodule segments a region of at least a third type of defect based on the partial image of the inspected component; The judgment module determines whether the inspected component is a defective component based on the area of ​​the at least first type of defect, the area of ​​the at least second type of defect, and the area of ​​the at least third type of defect.

16. The device of claim 15, wherein: The plurality of task detection submodules further include a third segmentation submodule and a fourth segmentation submodule, wherein the fourth segmentation submodule and the second segmentation submodule together constitute a combined segmentation submodule; The third segmentation submodule segments a region of at least a fourth type of defect based on the image of the inspected component; The combined segmentation submodule segments the region of the at least third type of defect based on the partial image of the inspected component; The judgment module further determines whether the inspected component is a defective component based on the area of ​​the at least fourth type of defect.

17. A system for automatic optical inspection (AOI), comprising: An image capture device for capturing an image of the inspected component; one or more processors; as well as One or more memories storing computer executable instructions that, when executed by the one or more processors, perform the operations of any one of claims 1 to 13.

18. A computer system comprising: one or more processors; as well as One or more memories storing computer executable instructions that, when executed by the one or more processors, perform the operations of any one of claims 1 to 13.

19. A machine-readable storage medium storing executable instructions which, when executed, cause one or more processors to perform the method of any one of claims 1 to 13.

Images