Method and apparatus for automated optical detection

Abstract

The invention relates to a method for AOI. The method comprises the steps that an image of a detected part of a first domain is obtained; inputting the image of the detected part into a detection model, wherein the detection model comprises a general feature extraction sub-model and at least one task detection sub-model related to a specific AOI task of the first domain; generating, by the general feature extraction sub-model, a first feature representation of the image based on the image of the detected component; at least one detection result related to a particular AOI task of the first domain is generated by the at least one task detection sub-model based on the first feature representation of the image.

Description

Technical Field

[0001] The present application relates to optical inspection, and more specifically, to a method and apparatus for multi-domain automatic optical inspection (AOI) based on a vision-based model. Background Art

[0002] During the manufacturing process of industrial products such as mechanical or electronic products, defects such as pits, wear, scratches, etc. may occur due to a variety of factors such as processing defects, imperfect manufacturing conditions, unexpected mechanical failures, etc. These defects increase manufacturing costs, shorten the service life of the finished product, and even cause substantial harm to the user. For example, defects in key mechanical or electronic components of automobiles can be potential safety issues. Therefore, detecting product defects is a key task to improve product quality.

[0003] Compared with manual inspection, automatic defect detection technology has obvious advantages. It can not only reduce the workload of inspectors, but also improve the efficiency, stability and performance of the inspection process, while adapting to various working environments that are not suitable for humans. The AOI system based on deep learning can achieve efficient and stable automatic inspection of industrial parts.

[0004] In the existing deep learning-based AOI system, a specific deep learning-based AOI detection model needs to be trained for industrial components of a domain (for example, a type of industrial component, or an industrial component production line). In order to train the detection model for AOI detection tasks, a large number of component images of the domain need to be collected and annotated as training data. However, especially for large enterprises with many types of industrial component production lines, for each type of industrial component AOI task, a large amount of image data of industrial components of this type must be collected and annotated, which requires a lot of collection and annotation work. In addition, for a new industrial component, such as a newly established industrial component production line, it takes a long time to collect a large number of defective component images, which requires more annotation time. In addition, training a specific deep learning-based AOI detection model for each domain also means that a large amount of computing resources and time costs are required for model training itself. At least based on the above problems, for AOI detection of industrial components with many types, it is necessary to further improve the development, maintenance, and upgrade efficiency of AOI systems based on deep learning. Summary of the invention

[0005] The following summary is provided to introduce selected concepts in a simplified form that are further described in the detailed description that follows. This summary is not intended to highlight key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] In response to the above problems, the present application provides an AOI detection method, device and system based on a visual basic model and a lightweight AOI task detection model. The inter-domain adaptive capability achieved by the visual basic model and the lightweight AOI task detection model effectively solves the above-mentioned efficiency problem of AOI detection of many types of industrial components.

[0007] According to one aspect of the present application, a method for AOI is provided, comprising: obtaining an image of a component to be inspected in a first domain, the first domain corresponding to a first industrial component type or to a first industrial component production line; inputting the image of the component to be inspected into a detection model, the detection model comprising a general feature extraction submodel and at least one task detection submodel related to a specific AOI task of the first domain; generating, by the general feature extraction submodel, a first feature representation of the image based on the image of the component to be inspected; generating, by the at least one task detection submodel, at least one detection result related to the specific AOI task of the first domain based on the first feature representation of the image.

[0008] According to one aspect of the present application, a method for training an inspection model for AOI is provided, comprising: collecting images of industrial components in multiple domains as a multi-domain AOI training dataset, wherein the multiple domains correspond to multiple industrial component types or multiple industrial component production lines respectively; training a general feature extraction submodel in the inspection model based on the multi-domain AOI training dataset to obtain a trained general feature extraction submodel; training at least one task detection submodel in the inspection model related to a specific AOI task in the first domain based on images of industrial components in the first domain as a single-domain AOI training dataset, wherein the first domain corresponds to a first industrial component type or to a first industrial component production line, wherein the trained general feature extraction submodel receives an image in the single-domain AOI training dataset and outputs a first feature representation of the image, and the at least one task detection submodel receives the first feature representation of the image and predicts at least one inspection result related to the specific AOI task in the first domain.

[0009] According to one aspect of the present application, a device for AOI is provided, comprising: an image acquisition module, used to obtain an image of a component to be inspected in a first domain, the first domain corresponding to a first industrial component type or to a first industrial component production line; a detection module, the detection model comprising a general feature extraction submodule and at least one task detection submodule related to a specific AOI task in the first domain, the general feature extraction submodule generating a first feature representation of the image based on the image of the component to be inspected, and the at least one task detection submodule generating at least one detection result related to the specific AOI task in the first domain based on the first feature representation of the image.

[0010] According to one aspect of the present application, a system for AOI is provided, comprising: an image capture device for capturing an image of an inspected component; one or more processors; and one or more memories, wherein the memories store computer executable instructions, and when the instructions are executed by the one or more processors, the operations described in the embodiments of the present application are performed.

[0011] According to one aspect of the present application, a computer system for AOI is provided, comprising: one or more processors; and one or more memories, wherein the memories store computer executable instructions, and when the instructions are executed by the one or more processors, the operations described in the embodiments of the present application are performed.

[0012] According to one aspect of the present application, a machine-readable storage medium is provided, which stores executable instructions. When the instructions are executed, one or more processors perform operations according to the embodiments of the present application.

[0013] By using the technical solution of the present application, a universal visual representation (UVR) of industrial component images adapted to multiple domains can be extracted by using a visual basic model trained based on industrial component images of multiple domains as an unlabeled data set. The UVR of the image, as an integrated rich feature representation, can be adapted to multiple domains corresponding to the training data set and multiple unknown domains outside the training data set, so that when used in conjunction with a lightweight task detection head of an AOI detection system, the task detection head can be trained based on a small amount of domain-specific labeled training data, thereby greatly improving the generalization and learning efficiency of the AOI detection system for new tasks. In addition, by integrating the visual basic model and multiple task detection heads dedicated to multiple domains for training, the visual basic model that has been pre-trained with a large number of industrial component images of multiple domains can be efficiently utilized, unlike the prior art that requires a specific deep learning-based AOI detection model to be trained for each domain. In addition, the visual basic model and multiple task detection heads dedicated to multiple domains can be integrated to perform detection, and the AOI detection processing of multiple production lines can be processed in a pipeline manner through the integrated AOI detection model, thereby further improving the utilization of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] A further understanding of the nature and advantages of the present application may be achieved by referring to the following drawings, in which similar components or features may have the same reference numerals.

[0015] Figure 1 A block diagram of an AOI system according to one embodiment is shown.

[0016] Figures 2A to 2D1 and 2 show block diagrams of an AOI apparatus for industrial components according to an embodiment.

[0017] Figure 3 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0018] Figure 4 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0019] Figure 5 A block diagram of an AOI system for industrial components is shown according to one embodiment.

[0020] Figure 6 A schematic diagram of a contrastive learning method for training a visual base model according to one embodiment is shown.

[0021] Figure 7 A schematic diagram of a MAE learning method for training a visual base model according to one embodiment is shown.

[0022] Figure 8 A schematic diagram for training a task detection model according to one embodiment is shown.

[0023] Fig. 9 A flow chart of a method for performing AOI according to one embodiment is shown.

[0024] Fig.10 A flowchart of a method for training an inspection model for AOI according to one embodiment is shown.

[0025] Fig.11 A block diagram of an apparatus for AOI of industrial components according to one embodiment is shown.

[0026] Fig.12 A block diagram of a processing system for AOI of industrial parts is shown according to one embodiment. DETAILED DESCRIPTION

[0027] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed can be changed without departing from the scope of protection of the present application. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0028] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0029] Figure 1 A block diagram of an AOI system according to one embodiment is shown.

[0030] The AOI system 100 may be a device for inspecting finished or semi-finished products of industrial parts produced. For example, the AOI system 100 may be deployed on a production line to inspect parts during the manufacturing process. The AOI system 100 may be part of a production line, or may be deployed at a location separate from the production line.

[0031] The AOI system 100 may include a front end 110, which may also be referred to as an optical system or a vision system 110. The front end 110 includes an imaging unit 1110 and an object carrier unit 1120. The imaging unit 1110 may capture an image of an object 10 placed on the carrier unit 1120. In one example, the imaging unit 1110 may include a sensing element and an optical focusing system that cooperate to achieve imaging of the object 10.

[0032] Examples of the carrier unit 1120 may be a mechanical platform, a mechanical arm, a conveyor belt, etc., which may grab, hold, and convey the object 10 so that the imaging unit 1110 may capture an image of the object 10. In one example, the imaging unit 1110 and the carrier unit 1120 may cooperate to capture multiple images of different parts of the object 10.

[0033] Although the carrier unit 1120 is shown as part of the front end 110, it should be understood that in some embodiments, the front end 110 may not actually include the carrier unit 1120, but rather the carrier unit is part of the production line. For example, the object 10 may be a semi-finished component placed on the carrier unit 1120 along the production line, and the imaging unit 1110 is deployed at this part of the production line to inspect the semi-finished component during manufacturing.

[0034] The front end 110 may send an image of the object 10 to the back end 120, which may also be referred to as a processing system or computing system 120. The processing system 120 may be implemented in various ways, for example, the processing system 120 may include one or more processors and / or controllers and one or more memories, and the processors and / or controllers may execute software to perform various operations or functions, such as operations or functions according to various aspects of the present application.

[0035] The processing system 120 may receive image data from the imaging unit 1110 and perform various operations by analyzing the image data. Figure 1 In the example of, the processing system 120 may include an AOI detection module 1210, and the AOI detection module 1210 may perform AOI detection processing on the received image of the object 10. For example, the AOI detection module 1210 may classify the image of the object 10, and the result of the classification may indicate whether the object 10 is a qualified product. For another example, the AOI detection module 1210 may segment the image of the object 10, thereby outputting a mask and a category indicating a defective portion in the image. For another example, the AOI detection module 1210 may recognize the image of the object 10, thereby outputting a bounding box and a category indicating a defective portion in the image. It can be understood that the AOI detection module 1210 can perform one or more of the above tasks, and the AOI detection module 1210 can perform the detection task to predict whether the object 10 as a product is a defective product and / or what kind of defect it has.

[0036] It is understood that the AOI detection module 1210 can be implemented in various ways, for example, it can be implemented as a software module or function. It is understood that in addition to the module 1210, the computing system 120 can also include other modules. For example, the control module in the computing system 120 can be configured to control the operation of the imaging unit 1110 and the carrying unit 1120 so as to capture one or more images of the object 10, for example, the imaging unit 1110 and the carrying unit 1120 can cooperate under the control of the control module to capture multiple images of different parts of the object. It is understood that such a control module can also be implemented in the front end 110. It is understood that the computing system 120 can be co-located with the front end 110 in the AOI system 100, for example, in the housing of the AOI system 100, and can also be located at a location away from the front end 110, for example, the computing system 120 can be implemented on a server or cloud, and the image captured by the front end 110 can be transmitted to the computing system 120 via a wired or wireless communication connection.

[0037] Figures 2A to 2D 1 and 2 show block diagrams of an AOI apparatus for industrial components according to an embodiment.

[0038] like Figure 2A As shown, images I1A and I2A may be images of industrial components 10 on a first production line. Since the industrial components of the first production line are of the same type, the images of these industrial components belong to the same domain D1. In this document, the term domain corresponds to an industrial component type or a production line of an industrial component. After receiving images of the industrial component 10 (e.g., images I1A, I2A) from the imaging unit 1110, an AOI detection module 210A specific to the domain D1 may process images I1A and I2A to extract feature representations of the images and predict outputs 220A related to AOI tasks, such as classification results, segmentation results, and / or recognition results regarding the images. Those skilled in the art will appreciate that any suitable neural network model may be used to implement the above-mentioned classification tasks, segmentation tasks, and recognition tasks.

[0039] Similarly, Figures 2B to 2D In the embodiment, images I1B and I2B may be images of industrial components 10 on the second production line, belonging to domain D2, images I1C and I2C may be images of industrial components 10 on the third production line, belonging to domain D3, and images I1D and I2D may be images of industrial components 10 on the fourth production line, belonging to domain D4. The AOI detection module 210B specific to domain D2 may process images I1B and I2B to extract feature representations of the images and predict outputs 220B related to the AOI task, the AOI detection module 210C specific to domain D3 may process images I1C and I2C to extract feature representations of the images and predict outputs 220C related to the AOI task, and the AOI detection module 210D specific to domain D4 may process images I1D and I2D to extract feature representations of the images and predict outputs 220D related to the AOI task.

[0040] AOI detection modules 210A to 210D are all specific implementations of AOI detection module 1210. Since AOI detection modules 210A to 210D are all domain-specific AOI detection modules, a large amount of training image data needs to be collected and annotated in the corresponding domain during the training phase of AOI detection modules 210A to 210D. Specifically, a large amount of training image data is collected and annotated in domain D1 to train AOI detection module 210A, a large amount of training image data is collected and annotated in domain D2 to train AOI detection module 210B, and so on.

[0041] Figure 3 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0042] like Figure 3 As shown, images I1A, I2A to I1D, I2D can be Figures 2A to 2DThe AOI detection module including the visual basic model 310 and the task detection models 320A to 320D can be Figure 1 A specific implementation of the AOI detection module 1210 is shown.

[0043] The visual base model 310 may be a self-attention mechanism model. For example, the visual base model 310 may use a hierarchical visual transformer (e.g., Swin, CvTVision, CSwin, SAM, DINOv2, etc.) as a backbone encoder to extract feature representations of images. By pre-training the visual base model 310 based on an unlabeled dataset of industrial component images of multiple domains, the trained visual base model 310 can extract a universal visual representation (UVR) of the industrial component image that is adapted to multiple domains. The UVR of the image can be adapted to multiple domains corresponding to the training dataset and multiple unknown domains outside the training dataset. Therefore, the AOI detection module 1210 including the visual base model 310 can learn universal AOI capabilities across multiple domains.

[0044] Task detection models 320A to 320D are task detection models specific to domains D1 to D4, respectively. Figures 2A to 2D The task detection models 210A to 210D specific to domains D1 to D4 shown, and the task detection models 320A to 320D are lightweight task detection models. Taking the task detection model 320A as an example, after the visual base model 310 receives the image I1A and generates the feature representation FM-A of the image I1A, the task detection model 320A performs the AOI detection task based on the feature representation FM-A and outputs the detection result 330A. For example, the AOI detection task can be one or more of a classification task, a segmentation task, and a recognition task, and the corresponding detection result 330A can be one or more of a classification result, a segmentation result, and a recognition result. Similarly, after the visual base model 310 receives the image I1B and generates the feature representation FM-B of the image I1B, the task detection model 320B performs the AOI detection task based on the feature representation FM-B and outputs the detection result 330B, and the task detection models 320C and 320D also perform the AOI detection task based on the feature representations FM-C and FM-D of the images I1C and I1D generated by the visual base model 310, respectively.

[0045] Since the pre-trained visual basic model 310 has already performed universal feature extraction on the component images across multiple domains, the task detection models 320A to 320D only need to be implemented using lightweight task models. Figures 2A to 2DCompared with the training of the task detection models 210A to 210D shown in the figure, only a small amount of domain-specific annotated training data is needed to train the task detection models 320A to 320D, thereby greatly improving the learning efficiency of the AOI detection system for new tasks. In addition, by integrating the visual base model and multiple task detection heads dedicated to multiple domains for training, the visual base model that has been pre-trained with a large number of industrial component images in multiple domains can be efficiently utilized, unlike the prior art that requires training a specific deep learning-based AOI detection model for each domain.

[0046] Figure 4 A block diagram of an AOI apparatus for industrial components according to one embodiment is shown.

[0047] Figure 4 Zhongyu Figure 3 The same reference numerals as shown in the drawings represent the same units, and the corresponding parts will not be described in detail again. Figure 4 As shown, the task detection models 320A to 320D include classification models 3210A to 3210D, segmentation models 3220A to 3220D, and recognition models 3230A to 3230D, respectively. Taking the task detection model 320A as an example, after the visual base model 310 receives the image I1A and generates the feature representation FM-A of the image I1A, the classification model 3210A in the task detection model 320A performs the classification task based on the feature representation FM-A and outputs the classification result 3310A, the segmentation model 3220A performs the segmentation task based on the feature representation FM-A and outputs the segmentation result 3320A, and the recognition model 3230A performs the recognition task based on the feature representation FM-A and outputs the recognition result 3330A. Similarly, after the visual base model 310 receives the image I1B and generates the feature representation FM-B of the image I1B, the classification model 3210B, the segmentation model 3220B and the recognition model 3230B in the task detection model 320B respectively perform the classification task, the segmentation task and the recognition task based on the feature representation FM-B to determine whether the component corresponding to the image is an unqualified component. The classification models 3210C and 3210D, the segmentation models 3220C and 3220D, and the recognition models 3230C and 3230D in the task detection models 320C and 320D also respectively perform the corresponding classification task, segmentation task and recognition task based on the feature representations FM-C and FM-D of the images I1C and I1D generated by the visual base model 310 to determine whether the component corresponding to the image is an unqualified component. Those skilled in the art will appreciate that although in Figure 4In the illustrated embodiment, the task detection models 320A to 320D all include a classification model, a segmentation model, and a recognition model. However, in other embodiments, one or more of the task detection models 320A to 320D may not include all of the classification models, segmentation models, and recognition models, but may only include one or more of the classification models, segmentation models, and recognition models, or may include other models other than the classification models, segmentation models, and recognition models. Those skilled in the art will appreciate that although in Figure 3 and 4 In the illustrated embodiment, the AOI detection module 1210 includes a visual base model 310 and a plurality of domain-specific task detection models 320 , but in other embodiments, the AOI detection module 1210 may include a visual base model 310 and one domain-specific task detection model 320 .

[0048] Figure 5 A block diagram of an AOI system for industrial components is shown according to one embodiment.

[0049] Figure 5 Zhongyu Figure 1 , 3 , 4 show the same or similar reference numerals to indicate the same or similar units, and the corresponding parts will not be described in detail again. Figure 5 As shown, each of the plurality of front ends 110A to 110D is connected to Figure 1 The front end 110 shown has a similar structure. For example, the front ends 110A to 110D are respectively arranged in a plurality of different production lines, and use imaging devices 1110A to 1110D to capture images of industrial components on the corresponding production lines, such as Figure 3 and 4 The AOI detection module 1210 on the processing system 120 includes the above-mentioned image I1A to I2D. Figure 3 and 4 The visual basis model 310 and task detection models 320A to 320D are described. The visual basis model 310 processes images I1A to I1D of multiple domains from the front ends 110A to 110D to obtain feature representations FM-A to FM-D of the images, and inputs the feature representations FM-A to FM-D to the task detection models 320A to 320D, respectively. The task detection models 320A to 320D obtain detection results 3310A to 3330A, 3310B to 3330B, 3310C to 3330C, and 3310D to 3330D based on the feature representations FM-A to FM-D, respectively. Figure 5In the embodiment of the AOI system shown, by integrating a visual base model and multiple task detection heads dedicated to multiple domains to perform AOI detection tasks for multiple production lines, the AOI detection processing of multiple production lines can be processed in a pipeline manner through the integrated AOI detection model, thereby potentially further improving the utilization rate of the computing resources of the computing device 120.

[0050] Those skilled in the art will appreciate that although Figure 5 In the illustrated embodiment, the AOI detection module 1210 includes a visual base model 310 and a plurality of domain-specific task detection models 320A to 320D, but in other embodiments, the AOI detection module 1210 may also include a visual base model 310 and a task detection model 320, which is used to process AOI detection of industrial parts from a domain (e.g., a production line).

[0051] Figure 6 A schematic diagram of a contrastive learning method for training a visual base model according to one embodiment is shown.

[0052] In order to train the visual base model 310, it is necessary to establish a large-scale dataset of industrial component images including multiple domains. Figure 5 In the example of the AOI system 500 shown, for a large enterprise with many kinds of industrial component production lines, the imaging unit 1110 deployed on each production line can obtain images of multiple kinds of industrial components as training data sets for pre-training the visual base model. For example, in one embodiment, millions of industrial component images can be collected from multiple production lines as AOI training data sets for pre-training the visual base model.

[0053] In one embodiment, it is possible to use Figure 6 The contrastive learning method shown is used to train the visual base model 310 to learn to extract feature representations of industrial component images of multiple domains. Figure 6 Taking the industrial component image I1 as an example, the data enhancement module 610 performs enhancement processing on the image I1 to obtain an image pair (I1-1, I1-2). For example, the data enhancement module 610 can obtain the image pair (I1-1, I1-2) by performing color change, brightness change, random shearing, flipping, rotation, etc. on the image I1, and the image pair can be called a positive pair.

[0054] The visual basis model 310 encodes the images I1-1 and I1-2 respectively to obtain the feature representations FM-1 and FM-2 of the corresponding images. The comparison module 620 obtains the similarity between the two images I1-1 and I1-2 based on the feature representations FM-1 and FM-2. For example, the similarity can be the cosine similarity between the feature vectors corresponding to the two images I1-1 and I1-2. In one example, the comparison module 620 can calculate the cosine similarity between the feature representations FM-1 and FM-2 as the similarity S1 between the two images I1-1 and I1-2. In another example, the comparison module 620 can also include an additional neural network layer for converting FM-1 and FM-2 to feature representations FM-1Z and FM-2Z in another space, and the comparison module 620 calculates the cosine similarity between the feature representations FM-1Z and FM-2Z as the similarity S1 between the two images I1-1 and I1-2.

[0055] The loss determination module 630 can determine the loss value L1 based on the similarity S1 between the two images I1-1 and I1-2. For example, the loss value L1 can be determined based on the similarity of the positive pair (I1-1, I1-2) and the similarity of the enhanced image and the other images in a batch constituting a negative pair. Determining the loss value based on the similarity of the positive pair and the negative pair is a commonly used technical means in the field of image processing, so it will not be described in detail. The visual base model 310 can then be updated based on the comparative learning loss value L1. For example, the known AdamW optimizer can be used to update the visual base model 310 based on the loss L1.

[0056] Figure 7 A schematic diagram of a masked autoencoder (MAE) learning method for training a visual basis model according to one embodiment is shown.

[0057] In one embodiment, it is possible to use Figure 7 The MAE method shown is used to train the visual base model 310 to learn to extract feature representations of industrial component images in multiple domains. Figure 7 Taking the industrial component image I1 as an example, the masking module 710 performs masking processing on the image I1 to obtain a masked image I1-M. For example, the masking module 710 divides the image I1 into small blocks, and then randomly selects a portion of the small blocks to retain, and masks all the remaining small blocks to obtain the masked image I1-M.

[0058] The visual basis model 310 encodes the masked image I1-M to obtain the feature representation FM-M of the masked image. The decoder 720 decodes the restored image I1-R based on the feature representation FM-M. The loss determination module 730 determines the MAE loss value L2 based on the restored image I1-R and the original image I1. For example, the loss determination module 730 can use the mean square error (MSE) function as the loss function to determine the MAE loss value L2, that is, the original pixels in the original image I1 and the reconstructed pixels in the restored image I1-R are subtracted and then the square sum is calculated. Then, the visual basis model 310 can be updated based on the MAE loss value L2. For example, the known AdamW optimizer can be used to update the visual basis model 310 based on the loss L2.

[0059] In one embodiment, both Figure 6 and 7 The contrastive learning method and MAE method shown are used to train the visual base model 310. For example, the weighted sum of the loss values ​​L1 and L2 can be used as the loss value to update the basic visual module.

[0060] Those skilled in the art will appreciate that any appropriate training method may be used to train the visual base model 310. The visual base model 310, which is trained with a large-scale dataset of industrial component images of multiple domains, has learned both domain-specific feature representations and general feature representations covering various domains. Therefore, the feature representations of industrial component images extracted by the visual base model 310 as a feature extractor can help the downstream domain-specific AOI detection modules 320 to learn AOI detection tasks adapted to various domains through a lightweight domain-specific adaptive training process.

[0061] Figure 8 A schematic diagram for training a task detection model 320 according to one embodiment is shown.

[0062] by Figure 3 and 4Taking the task detection model 320C specific to the domain D3 as an example, the task detection model 320 may include one or more of a classification model 3210, a segmentation model 3220, and a recognition model 3230. In one embodiment, the classification model 3210 may be a multi-layer perceptron (MLP) model, the segmentation model 3220 may be a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism, and the recognition model 3230 may be a bounding box decoder with classification results of a CNN model or a decoder based on an attention mechanism. The task detection model 320 may be trained based on a set of images collected and annotated in the domain D3 as a training data set, and more specifically, the classification model 3210, the segmentation model 3220, and / or the recognition model 3230 therein may be trained. In one embodiment, the lightweight task heads 3210, 3220, and 3230 may be trained based on a small amount of training data sets specific to the domain D3, for example, the small amount of training data sets may be a data set in units of hundreds of images.

[0063] The visual basis model 310 encodes the image I1 to obtain the feature representation FM of the image. The task detection model 320 obtains a prediction P about the detection task based on the feature representation. For example, the classification model 3210, the segmentation model 3220 and / or the recognition model 3230 respectively obtain predictions P1, P2 and / or P3 about the detection task based on the feature representation. The loss determination module 830 determines the loss value L3 based on the prediction results P1, P2 and / or P3 and the corresponding labels representing the annotated detection results, specifically including loss values ​​L3-1, L3-2 and / or L3-3 corresponding to the prediction results P1, P2 and / or P3, respectively. Various appropriate methods for determining the supervised loss using the prediction value and the label value can be used to determine the loss value L3. Then, the task detection model 320 can be updated based on the supervised loss value L3, specifically, the classification model 3210, the segmentation model 3220 and / or the recognition model 3230 can be updated based on the loss values ​​L3-1, L3-2 and / or L3-3. For example, the known AdamW optimizer can be used to update the task detection model 320 based on the loss L3. In one embodiment, the parameters of the pre-trained visual base model 310 are frozen during the process of updating the task detection model 320. In another embodiment, a small number of parameters of the pre-trained visual base model 310 can be updated simultaneously during the process of updating the task detection model 320.

[0064] exist Figure 8 In the illustrated embodiment, the task detection model 320 may be trained using industrial component images of domains other than the industrial component images of multiple domains used to train the visual base model 310. Figure 3The domains D1 to D4 shown represent multiple domains for training the visual base model 310, and the domain D5 represents the domain for training the task detection model 320. Those skilled in the art will appreciate that the number of domains for training the visual base model 310 is actually greater. The visual base model 310 trained with a large-scale dataset of industrial component images including multiple domains D1 to D4 not only learns domain-specific and universal feature representations covering each domain, but also the universal feature representations can be transferred to domains that have not been seen by the visual base model 310. Therefore, in this embodiment, the task detection model 320 corresponding to the domain D5 that has not been seen by the visual base model 310 can be trained. This embodiment is particularly advantageous in performing AOI of produced components at the initial stage of establishing a new production line corresponding to the domain D5, for example.

[0065] Fig. 9 A flow chart of a method for performing AOI according to one embodiment is shown.

[0066] At step 910 , an image of an inspected component of a first domain is obtained, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line.

[0067] At step 920, the image of the inspected component is input into an inspection model, which includes a general feature extraction sub-model and at least one task detection sub-model related to a specific AOI task of the first domain.

[0068] At step 930, the general feature extraction sub-model generates a first feature representation of the image based on the image of the inspected component.

[0069] At step 940 , at least one detection result related to the specific AOI task of the first domain is generated by the at least one task detection sub-model based on the first feature representation of the image.

[0070] In one embodiment, the specific AOI task of the first domain includes at least one or at least two of the following tasks: a classification task of classifying the image of the inspected component into one of multiple categories, for example, the multiple categories include normal, defective or others; a segmentation task of segmenting the defective area in the image of the inspected component, for example, the segmented target area is represented by a mask, wherein the pixel value of the target area is 1 and the pixel value of the remaining area is 0; and a recognition task of identifying the defective area in the image of the inspected component, for example, the identified area is represented by a bounding box.

[0071] In one embodiment, when the specific AOI task of the first domain is the classification task, the task detection submodel is an MLP model; when the specific AOI task of the first domain is the segmentation task, the task detection submodel is a mask decoder of a CNN model or a decoder based on an attention mechanism; and when the specific AOI task of the first domain is the recognition task, the task detection submodel is a bounding box decoder with classification results of a CNN model or a decoder based on an attention mechanism.

[0072] In one embodiment, the general feature extraction sub-model is a visual base model based on the attention mechanism.

[0073] In one embodiment, the general feature extraction sub-model is trained based on images of industrial components in multiple domains, wherein the multiple domains correspond to multiple industrial component types or multiple industrial component production lines, respectively.

[0074] In one embodiment, the first domain is one of the plurality of domains. In one embodiment, the first domain is a domain outside of the plurality of domains. The task detection submodel is trained based on images of industrial components in the first domain. In one embodiment, the first domain images used to train the task detection submodel are different from, or not identical to, the first domain images used to train the general feature extraction submodel.

[0075] In one embodiment, the method 900 also includes obtaining a second image of a second inspected component in a second domain, the second domain corresponding to a second industrial component type or corresponding to a second industrial component production line; inputting the second image of the second inspected component into the detection model, the detection model also including at least one second task detection sub-model related to a specific AOI task of the second domain; generating a first feature representation of the second image based on the second image of the second inspected component by the general feature extraction sub-model; generating at least one detection result related to the specific AOI task of the second domain based on the first feature representation of the second image by the at least one second task detection sub-model.

[0076] Fig.10 A flowchart of a method for training an inspection model for AOI according to one embodiment is shown.

[0077] In step 1010 , images of industrial components in multiple domains are collected as a multi-domain AOI training dataset, wherein the multiple domains correspond to multiple industrial component types or multiple industrial component production lines, respectively.

[0078] In step 1020, a general feature extraction sub-model in the detection model is pre-trained based on the multi-domain AOI training data set to obtain a trained general feature extraction sub-model.

[0079] In step 1030, at least one task detection sub-model related to a specific AOI task in the first domain is trained in the detection model based on images of industrial components in the first domain as a single-domain AOI training dataset, and the first domain corresponds to a first industrial component type or to a first industrial component production line, wherein the trained general feature extraction sub-model receives the image in the single-domain AOI training dataset and outputs a first feature representation of the image, and the at least one task detection sub-model receives the first feature representation of the image and predicts at least one detection result related to the specific AOI task in the first domain.

[0080] In one embodiment, in step 1020, data enhancement is performed on the unlabeled multi-domain AOI training dataset to obtain a set of multi-domain AOI training image pairs, and the general feature extraction sub-model is trained using a contrastive learning method based on the multi-domain AOI training image pair set; and / or, the general feature extraction sub-model is trained using a MAE method based on the unlabeled multi-domain AOI training dataset.

[0081] In one embodiment, at step 1020, the training of the general feature extraction submodel using the contrastive learning method based on the multi-domain AOI training image pair set includes: the general feature extraction submodel receives an image pair in the multi-domain AOI training image pair set and outputs a first feature representation of the image pair, the contrastive learning module obtains a similarity of the image pair based on the first feature representation of the image pair, obtains a contrastive learning loss based on the similarity of the image pair, and updates the general feature extraction submodel based on the contrastive learning loss; and / or, the training of the general feature extraction submodel using the MAE method includes: randomly masking an original image in the multi-domain AOI training dataset to obtain a masked image, the general feature extraction submodel receives the masked image and outputs a first feature representation of the masked image, the decoder module predicts a restored image based on the first feature representation of the masked image, obtains a MAE loss based on the restored image and the original image, and updates the general feature extraction submodel based on the MAE loss.

[0082] In one embodiment, at step 1030 , the task detection sub-model is trained based on the labeled single-domain AOI training dataset.

[0083] In one embodiment, the method 1000 further includes: training at least one second task detection sub-model in the detection model related to a second specific AOI task in the second domain based on an image of an industrial component in the second domain as a second single-domain AOI training data set, the second domain corresponding to a second industrial component type or corresponding to a second industrial component production line, wherein the trained general feature extraction sub-model receives a second image in the second single-domain AOI training data set and outputs a first feature representation of the second image, and the at least one second task detection sub-model receives the first feature representation of the second image and predicts at least one detection result related to the second specific AOI task in the second domain.

[0084] Fig.11 A block diagram of an apparatus for AOI of industrial components according to an embodiment is shown. The apparatus 1100 includes: an image acquisition module 1110, which is used to obtain an image of a component to be inspected in a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; a detection module 1120, the detection model includes a general feature extraction submodule and at least one task detection submodule related to a specific AOI task of the first domain, the general feature extraction submodule generates a first feature representation of the image based on the image of the component to be inspected, and the at least one task detection submodule generates at least one detection result related to the specific AOI task of the first domain based on the first feature representation of the image.

[0085] In one embodiment, the image acquisition module 1110 obtains a second image of a second inspected component in a second domain, and the second domain corresponds to a second industrial component type or corresponds to a second industrial component production line, wherein the detection module 1120 also includes at least one second task detection submodule related to a specific AOI task of the second domain, wherein the general feature extraction submodule generates a first feature representation of the second image based on the second image of the second inspected component, and the at least one second task detection submodule generates at least one detection result related to the specific AOI task of the second domain based on the first feature representation of the second image.

[0086] In one embodiment, Fig.11 The detection module 1120 shown can be a combination of the above Figure 1 , 3 , 4, 5 described detection model 1210, wherein the general feature extraction submodule and the task detection submodule can be respectively the above combined Figure 3 , 4 , the visual basis model 310 and the task detection model 320 described in 5.

[0087] Fig.121 shows a block diagram of a processing system for AOI of industrial components according to one embodiment. The control system or processing system 1200 may include one or more control units or processing units 1210 that execute one or more machine-readable instructions stored or encoded in a machine-readable storage medium (i.e., memory 1220). Although not described in detail in the Fig.12 However, those skilled in the art will appreciate that the control system 1200 may include various other components, such as various communication modules, bus modules, and possible user interface modules. In one embodiment, the processing unit 1210 is configured to execute the above combined Figure 1-11 Describes the various operations and functions.

[0088] According to one embodiment, a program product such as a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium may have instructions, which, when executed by the processing unit 1210, can execute the above combined with the various embodiments of the present application. Figure 1-11 Describes the various operations and functions.

[0089] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "example" used throughout this specification may mean "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0090] The above description of the present application is provided to enable any person of ordinary skill in the art to implement or use the present application. It is obvious to a person of ordinary skill in the art that various modifications are made to the present application, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present application. Therefore, the present application is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for automatic optical inspection (AOI), comprising: Obtaining an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; Inputting the image of the inspected component into an inspection model, wherein the inspection model includes a general feature extraction sub-model and at least one task detection sub-model related to a specific AOI task of the first domain; generating, by the general feature extraction sub-model, a first feature representation of the image based on the image of the detected component; At least one detection result related to a specific AOI task in the first domain is generated by the at least one task detection sub-model based on the first feature representation of the image.

2. The method of claim 1, wherein: The specific AOI tasks of the first domain include at least one or at least two of the following tasks: A classification task of classifying the image of the detected component into one of a plurality of categories; A segmentation task of segmenting a defective area in the image of the inspected component; as well as The recognition task is to identify the defective area in the image of the inspected component.

3. The method of claim 2, wherein: When the specific AOI task of the first domain is the classification task, the task detection sub-model is a multi-layer perceptron (MLP) model; When the specific AOI task of the first domain is the segmentation task, the task detection sub-model is a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism; as well as When the specific AOI task of the first domain is the recognition task, the task detection sub-model is a bounding box decoder with classification results of a CNN model or a decoder based on an attention mechanism.

4. The method according to claim 1, wherein: The general feature extraction sub-model is a visual base model based on the attention mechanism.

5. The method of claim 4, wherein: The general feature extraction sub-model is trained based on images of industrial components in multiple domains, wherein the multiple domains correspond to multiple industrial component types or multiple industrial component production lines, respectively.

6. The method of claim 5, wherein: The first domain is one of the multiple domains, or the first domain is a domain outside the multiple domains, The task detection sub-model is trained based on images of industrial parts in the first domain.

7. The method of claim 1, further comprising: obtaining a second image of a second inspected component in a second domain, the second domain corresponding to a second industrial component type or corresponding to a second industrial component production line; Inputting a second image of the second inspected component into the inspection model, wherein the inspection model further includes at least one second task inspection sub-model related to a specific AOI task of the second domain; generating, by the general feature extraction sub-model, a first feature representation of the second image based on the second image of the second detected component; At least one detection result related to a specific AOI task in the second domain is generated by the at least one second task detection sub-model based on the first feature representation of the second image.

8. A method for training an inspection model for automatic optical inspection (AOI), comprising: Collecting images of industrial components in multiple domains as a multi-domain AOI training dataset, wherein the multiple domains correspond to multiple industrial component types or multiple industrial component production lines respectively; Training the universal feature extraction sub-model in the detection model based on the multi-domain AOI training data set to obtain a trained universal feature extraction sub-model; At least one task detection sub-model related to a specific AOI task in a first domain is trained in the detection model based on images of industrial components in the first domain as a single-domain AOI training dataset, and the first domain corresponds to a first industrial component type or to a first industrial component production line, wherein the trained general feature extraction sub-model receives images in the single-domain AOI training dataset and outputs a first feature representation of the image, and the at least one task detection sub-model receives the first feature representation of the image and predicts at least one detection result related to the specific AOI task in the first domain.

9. The method of claim 8, wherein: The training of the general feature extraction sub-model in the detection model based on the multi-domain AOI training data set includes: Performing data enhancement on the unlabeled multi-domain AOI training dataset to obtain a multi-domain AOI training image pair set, and training the general feature extraction sub-model using a contrastive learning method based on the multi-domain AOI training image pair set; and / or Based on the unlabeled multi-domain AOI training dataset, the general feature extraction sub-model is trained using a masked autoencoder (MAE) method.

10. The method according to claim 9, in, The training of the general feature extraction sub-model using a contrastive learning method based on the multi-domain AOI training image pair set includes: the general feature extraction sub-model receives an image pair in the multi-domain AOI training image pair set and outputs a first feature representation of the image pair, a contrastive learning module obtains a similarity of the image pair based on the first feature representation of the image pair, obtains a contrastive learning loss based on the similarity of the image pair, and updates the general feature extraction sub-model based on the contrastive learning loss; and / or Among them, the use of the MAE method to train the general feature extraction sub-model includes: masking the original image in the multi-domain AOI training data set to obtain a masked image, the general feature extraction sub-model receiving the masked image and outputting a first feature representation of the masked image, the decoder module predicting a restored image based on the first feature representation of the masked image, obtaining an MAE loss based on the restored image and the original image, and updating the general feature extraction sub-model based on the MAE loss.

11. The method of claim 8, wherein: The training of at least one task detection sub-model related to a specific AOI task in the first domain in the detection model based on images of industrial parts in the first domain as a single-domain AOI training dataset includes: training the task detection sub-model based on the labeled single-domain AOI training dataset.

12. The method of claim 8, further comprising: At least one second task detection sub-model related to a second specific AOI task in the second domain in the detection model is trained based on images of industrial components in the second domain as a second single-domain AOI training dataset, and the second domain corresponds to a second industrial component type or corresponds to a second industrial component production line, wherein the trained general feature extraction sub-model receives a second image in the second single-domain AOI training dataset and outputs a first feature representation of the second image, and the at least one second task detection sub-model receives the first feature representation of the second image and predicts at least one detection result related to the second specific AOI task in the second domain.

13. An apparatus for automatic optical inspection (AOI), comprising: An image acquisition module, configured to acquire an image of a detected component of a first domain, wherein the first domain corresponds to a first industrial component type or corresponds to a first industrial component production line; A detection module, wherein the detection model includes a general feature extraction submodule and at least one task detection submodule related to a specific AOI task of the first domain, wherein the general feature extraction submodule generates a first feature representation of the image based on the image of the inspected component, and the at least one task detection submodule generates at least one detection result related to the specific AOI task of the first domain based on the first feature representation of the image.

14. The device of claim 13, wherein: The image acquisition module acquires a second image of a second detected component in a second domain, wherein the second domain corresponds to a second industrial component type or corresponds to a second industrial component production line. The detection module also includes at least one second task detection submodule related to the specific AOI task of the second domain, wherein the general feature extraction submodule generates a first feature representation of the second image based on the second image of the second inspected component, and the at least one second task detection submodule generates at least one detection result related to the specific AOI task of the second domain based on the first feature representation of the second image.

15. A system for automatic optical inspection (AOI), comprising: An image capture device for capturing an image of the inspected component; one or more processors; as well as One or more memories storing computer executable instructions that, when executed by the one or more processors, perform the operations of any one of claims 1 to 12.

16. A computer system comprising: one or more processors; as well as One or more memories storing computer executable instructions that, when executed by the one or more processors, perform the operations of any one of claims 1 to 12.

17. A machine-readable storage medium storing executable instructions which, when executed, cause one or more processors to perform the method of any one of claims 1 to 12.

Images