METHOD AND DEVICE FOR AUTOMATIC OPTICAL INSPECTION (AOI)

Abstract

The present application relates to a method for automatic optical inspection (AOI) comprising: obtaining an image of a captured component of a first area; inputting the image of the captured component into a capture model, wherein the capture model comprises a universal submodel for feature extraction and at least one submodel for task capture relating to a specific AOI task for that first area; a first feature representation of the image, generated from the image of the captured component by means of the submodel for universal feature extraction; generating at least one capture result, which, on the basis of this first feature representation of the image by this at least one submodel for task capture, relates to a specific AOI task for that first area.

Description

FIELD OF THE INVENTION

[0001] The present application relates to an optical inspection, specifically it relates to a method and a device for automatic optical inspection (AOI) based on a visual base model. STATE OF THE ART

[0002] During the manufacturing process of industrial products such as machinery or electronic products, defects such as pitting, wear, and scratches can occur due to various factors such as processing errors, imperfect manufacturing conditions, unexpected mechanical failures, and so on. These defects increase manufacturing costs, shorten the service life of finished products, and even cause significant harm to users. For example, defects in critical mechanical or electronic components of an automobile could pose a potential safety hazard. Therefore, detecting product defects is a key task for improving product quality.

[0003] Compared with manual inspection, automatic defect inspection technology not only reduces the workload of inspection personnel but also improves the efficiency, stability, and performance of the inspection process, while adapting to various working environments that are unsuitable for humans. The deep learning-based AOI system can achieve efficient and stable automatic detection of industrial components.

[0004] According to the state of the art, deep learning-based AOI systems require training a specific deep learning-based AOI inspection model for industrial components in a domain (e.g., a specific type of industrial component or an industrial component production line). This requires collecting and labeling a large number of component images in a domain as training data to train the inspection model for AOI inspection tasks. However, especially in large companies with production lines for many types of industrial components, a large amount of image data of this type of industrial component must be collected and labeled for each AOI task for industrial components, requiring significant collection and labeling effort.Furthermore, for a new industrial component, such as a newly established industrial component production line, it takes a long time to collect a large number of images of defective components, and labeling requires correspondingly more time. Furthermore, training a specific deep learning-based AOI detection model for each area also means that the model training itself requires a large amount of computing resources and time. Due to at least the above-mentioned problems, for AOI inspection of many types of industrial components, it is necessary to further improve the development, maintenance, and upgrade efficiency of AOI systems based on deep learning. DISCLOSURE OF THE INVENTION

[0005] The following introduction is provided to briefly introduce some of the selected concepts that will be further described in a later detailed description. The introduction is neither intended to emphasize essential or necessary features of the claimed subject matter nor to limit the scope of the claimed subject matter.

[0006] In response to the above-mentioned problems, the present application provides a method, an apparatus, and a system for AOI detection based on a visual base model and a lightweight AOI task detection model, wherein the cross-domain adaptability achieved by the visual base model and the lightweight AOI task detection model effectively solves the above-mentioned efficiency problem of AOI inspection of industrial components of many types.

[0007] According to one aspect of the present application, a method for AOI is provided, comprising: obtaining images of the detected component in a first region, wherein the first region corresponds to a first industrial component type or a first industrial component production line; inputting the image of the detected component into a detection model, wherein the detection model comprises a universal feature extraction sub-model and at least one task detection sub-model related to the specific AOI task of the first region; generating a first feature representation of the image based on the detected component by the universal feature extraction sub-model; generating at least one detection result related to a specific AOI task of the first region by the at least one task detection sub-model based on the first feature representation of an image.

[0008] According to one aspect of the present application, a method for training AOI detection models is provided, comprising: collecting industrial component images in multiple regions as a multi-region AOI training dataset, wherein the multiple regions correspond to multiple industrial component types or to multiple industrial component production lines, respectively; training a universal feature extraction submodel in the detection model based on the multi-region AOI training dataset to obtain a trained universal feature extraction submodel;Training at least one task detection submodel in the detection model related to a specific AOI task in the first region based on industrial component images in a first region corresponding to a first industrial component type or a first industrial component production line as a single-region AOI training dataset, wherein the trained universal feature extraction submodel receives the image in the single-region AOI training dataset and outputs the first feature representation of the image, wherein the at least one task detection submodel receives the first feature representation of the image and predicts at least one detection result related to a specific AOI task of the first region;

[0009] According to one aspect of the present application, an apparatus for AOI is provided, comprising: an image acquisition module used for acquiring an image of a detected component of a first region, the first region corresponding to a first industrial component type or a first industrial component production line;a detection module, wherein the detection model comprises a submodel for universal feature extraction and at least one submodel for task detection relating to a specific AOI task for said first region, wherein a first feature representation of the image is generated from the image of the detected component by means of the submodel for universal feature extraction, wherein the submodule for the at least one task detection generates at least one detection result relating to a specific AOI task of the first region based on the first feature representation of the image;

[0010] According to one aspect of the present application, a system for AOI is provided, comprising: an image capture device for capturing images of the captured components; one or more processors; and one or more memories, the memory storing computer-executable instructions, the instructions executed by the one or more processors performing the operations described in the embodiments of the present application.

[0011] According to one aspect of the present application, a computer system for AOI is provided, comprising: one or more processors and one or more memories storing computer-executable instructions that, when executed by the one or more processors, perform the operations according to the embodiments of the present application.

[0012] According to another aspect of the present invention, a machine-readable storage medium is provided that stores executable instructions that, when executed, cause one or more processors to perform the operations according to the embodiments of the present application.

[0013] Using the technical solution of this application, a universal visual representation (UVR) of industrial component images adapted to multiple regions can be extracted. Using the base visual model trained on industrial component images in multiple regions as an unlabeled dataset, the UVR, as an integrated feature-rich representation of this image, can adapt to multiple regions corresponding to the training dataset as well as multiple unknown regions outside the training dataset. Therefore, when used in conjunction with the lightweight task detection head of the AOI detection system, the task detection head can be trained based on a small amount of region-specific labeled training data, thereby greatly improving the generalization of the AOI detection system to new tasks.Furthermore, by integrating the base visual model and multiple task capture heads, each trained for multiple areas, the base visual model, which has been pre-trained with a large number of images of industrial components from multiple areas, can be efficiently utilized. Therefore, according to the state of the art, it is no longer necessary to train a specific deep learning-based AOI capture model for each area. Furthermore, capture can be performed by integrating a base visual model with multiple task capture heads, each assigned to multiple areas. DESCRIPTION OF THE CHARACTERS

[0014] A further understanding of the nature and advantages of the present application may be achieved by reference to the following figures. In the figures, similar components or features may have the same reference numerals. Fig. 1 shows a block diagram of an AOI system according to an embodiment. Fig. 2A to 2D each show a block diagram of an AOI device for industrial components according to an embodiment. Fig. 3 shows a block diagram of an AOI device for industrial components according to an embodiment. Fig. 4 shows a block diagram of an AOI device for industrial components according to an embodiment. Fig. 5 shows a block diagram of an AOI device for industrial components according to an embodiment. Fig. 6 shows a schematic representation of a contrastive learning method for training a visual base model according to an embodiment. Fig. 7 shows a schematic representation of an MAE learning method for training a visual base model according to an embodiment. Fig. 8 shows a schematic diagram for training a task detection model according to an embodiment. Fig. 9 shows a flowchart of a method for performing an AOI according to an embodiment. Fig. 10 shows a flowchart of a method for training a detection model for AOI according to an embodiment. Fig. 11 shows a block diagram of an apparatus for AOI of industrial components according to an embodiment. Fig. 12 shows a block diagram of a processing system for AOI of industrial components according to an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that the explanation of these embodiments is merely intended to facilitate understanding by those skilled in the art and to enable the subject matter of the present document to be practiced, without limiting the scope, applicability, or examples in the claims. The functions and arrangement of the described elements may be changed without departing from the scope of the content of the present application. For individual embodiments, it is conceivable to omit, replace, or add various operations or components as needed. For example, the described methods may be performed in a different order than described, and individual steps may be added, omitted, or combined.In addition, features described with respect to some examples may also be combined in other examples.

[0016] The term "comprising" and its variants used herein may represent an open-ended term and mean "including, but not limited to." The term "based on" may mean "based at least in part on." The terms "an embodiment," "some embodiments," etc., may mean "at least one embodiment." The term "another embodiment" means "at least one other embodiment." The terms "first," "second," etc., may refer to different or the same objects. Other definitions, either explicit or implicit, may be provided below. The definition of a term is consistent throughout this specification unless the context clearly indicates otherwise.

[0017] Fig. 1 shows a block diagram of an AOI system according to an embodiment.

[0018] The AOI system 100 may be a device for inspecting finished or semi-finished products of industrial components being manufactured. For example, the AOI system 100 may be deployed in a production line to inspect parts during the manufacturing process. The AOI system 100 may be part of the production line or deployed separately from the production line.

[0019] The AOI system 100 may include a front end 110, wherein the front end 110 may also be referred to as an optical system or visual system 110. The front end 110 includes an imaging unit 1110 and a slide unit 1120. The imaging unit 1110 may capture the image of an object 10 placed on the slide unit 1120. In one example, the imaging unit 1110 may include a sensor element and an optical focusing system that cooperate to image the object.

[0020] Examples of the carrier unit 1120 may be a mechanical platform, a robotic arm, a conveyor belt, etc., that can grasp, hold, and transport the object 10 so that the imaging unit 1110 captures an image of the object 10. In one example, the imaging unit 1110 and the carrier unit 1120 may cooperate to capture multiple images of different parts of the object 10.

[0021] Although the carrier unit 1120 is illustrated as part of the front end 110, it should be understood that in some embodiments, the front end 110 may not actually include the carrier unit 1120, but rather the carrier unit is part of the production line. For example, the object 10 may be a semi-finished component that is placed on the transport unit 1120 along the production line, and the imaging unit 1110 is deployed at this part of the production line to inspect the semi-finished component during manufacturing.

[0022] The front end 110 may send images of the object 10 to a back end 120, which may also be referred to as a processing system or computer system 120. The processing system 120 may be implemented in various ways, for example, with one or more processors and / or controllers and one or more memories, where the processor and / or controller may execute software to perform various operations or functions, e.g., operations or functions according to the various aspects of the present application.

[0023] The processing system 120 may receive image data from the imaging unit 1110 and perform various operations by analyzing the image data. In the example of Fig. 1, the processing system 120 may include an AOI detection module 1210, and the AOI detection module 1210 may perform AOI detection processing on the received image of the object 10. For example, the AOI detection module 1210 may classify the image of the object 10, where the result of the classification may indicate, for example, whether the object 10 is a defect-free product. As another example, the AOI detection module 1210 may perform segmentation processing on the image of the object 10, thereby outputting a mask and a category indicating a defect-free part in the image. As another example, the AOI detection module 1210 may perform identification processing on the image of the object 10, thereby outputting a bounding box and a category indicating a defect-free part in the image.It is understood that the AOI detection module 1210 may perform one or more of the above-mentioned tasks, wherein by performing the detection task of the AOI detection module 1210, it may be predicted whether the object 10 as a product is a defective product and / or what defects it has.

[0024] It is understood that the AOI acquisition module 1210 can be implemented in various ways, for example, as a software module or function. It is understood that the computer system 120 can also include other modules in addition to the module 1210. For example, a control module in the computer system 120 can be configured to control the operation of the imaging unit 1110 and the carrier unit 1120 to acquire one or more images of the object 10, wherein, for example, the imaging unit 1110 and the carrier unit 1120 can cooperate under the control of the control module to acquire multiple images of different parts of the object. It is understood that such a control module can also be implemented in the front end 110.It is understood that the computer system 120 may be located together with the front end 110 in the AOI system 100, for example in a housing of the AOI system 100, but may also be located remotely from the front end 110, for example, the computer system 120 may be implemented on a server or in a cloud, and images captured by the front end 110 may be transmitted to the computer system 120 via wired or wireless communication links.

[0025] Fig. 2A to 2D each show a block diagram of an AOI device for industrial components according to an embodiment.

[0026] As in Fig. 2A, images I1A and I2A may be images of the industrial component 10 in the first production line. Since the industrial components of the first production line are industrial components of the same type, the images of these industrial components belong to the same area D1. In the present text, the term area corresponds to an industrial component type or a production line of an industrial component. After receiving images of the industrial component 10 from the imaging unit 1110 (e.g., images I1A, I2A), the AOI detection module 210A specific to the area D1 may process the images I1A and I2A, extracting the feature representation of the image and predicting an output 220A related to the AOI task, for example, a classification result, a segmentation result, and / or an identification result related to the image.It is understandable for those skilled in the art that any suitable neural network model can be used to implement the above-mentioned classification tasks, segmentation tasks, and identification tasks.

[0027] Similarly, in the embodiments of Fig. 2B to 2D, images I1B and I2B may be images of industrial components 10 in a second production line belonging to area D2, images I1C and I2C may be images of industrial components 10 in a third production line belonging to area D3, and images I1D and I2D may be images of industrial components 10 in a fourth production line belonging to area D4.The AOI detection module 210B specific to area D2 may process the images I1B and I2B to extract feature representations of the images to predict the output 220B related to the AOI task, the AOI detection module 210C specific to area D3 may process the images I1C and I2C to extract feature representations of the images and predict the output 220C related to the AOI task, and the AOI detection module 210D specific to area D4 may process the images I1D and I2D to extract feature representations of the images and predict the output 220D related to the AOI task.

[0028] The AOI detection modules 210A to 210D are specific implementations of the AOI detection module 1210. Since the AOI detection modules 210A to 210D are area-specific AOI detection modules, a large amount of training image data must be collected and labeled in the corresponding area during the training phase of the AOI detection modules 210A to 210D. Specifically, a large amount of training image data is collected and labeled in area D1 to train the AOI detection module 210A, a large amount of training image data is collected and labeled in area D2 to train the AOI detection module 210B, and so on.

[0029] Fig. 3 shows a block diagram of an AOI device for industrial components according to an embodiment.

[0030] As in Fig. 3, images I1A, I2A to I1D and I2D may be industrial component images corresponding to several Fig. 2A to 2D. As mentioned above, images I1A, I2A to I1D, and I2D are, for example, images of industrial components from four different production lines each. The AOI acquisition module, including the base visual model 310 and the task acquisition models 320A to 320D, may be a specific implementation of the Fig. 1 shown AOI detection module 1210.

[0031] The visual base model 310 may be a self-attention mechanism model. For example, the visual base model 310 may use a hierarchical visual transformer (such as Swin, CvTVision, CSwin, SAM, DINOv2, etc.) as a backbone encoder to extract the features of the image. By pre-training the visual base model 310 based on an unlabeled dataset of industrial component images from multiple regions, the trained visual base model 310 can extract a universal visual representation (UVR) of the industrial component images adapted to multiple regions. The UVR of this image can be adapted to a plurality of regions corresponding to the training dataset, as well as to a plurality of unknown regions outside the training dataset, so that the AOI detection module 1210, which includes this visual base model 310, can learn a generalized AOI capability across multiple regions.

[0032] The task acquisition models 320A to 320D are task-specific acquisition models for the areas D1 to D4. In comparison to the task acquisition models 210A to 210D, which are for the Fig. 2A to 2D, the task detection models 320A to 320D are lightweight task detection models. For example, in the task detection model 320A, after the base visual model 310 receives the image I1A and generates the feature representation FM-A of the image I1A, the task detection model 320A performs the AOI detection task based on the feature representation FM-A and outputs the detection result 330A. For example, the AOI detection task may be one or more of classification, segmentation, and identification tasks, and the corresponding detection result 330A may be one or more of classification, segmentation, and identification results.Similarly, after the base visual model 310 receives the image I1B and generates the feature representation FM-B of the image I1B, the task detection model 320B executes the AOI detection task based on the feature representation FM-B and outputs the detection result 330B, and the task detection models 320C and 320D also execute the AOI detection task based on the feature representations FM-C and FM-D of the images I1C and I1D generated by the base visual model 310, respectively.

[0033] Since the pre-trained visual base model 310 has already performed universal feature extraction on the sub-images across multiple regions, the task detection models 320A to 320D only need to adopt lightweight task models to implement them. Accordingly, compared to the training of the Fig. 2A to 2D, only a small amount of area-specific labeled training data is required to train the task detection models 320A to 320D, thereby significantly improving the learning efficiency of the AOI detection system for new tasks. Furthermore, by integrating the base visual model and multiple task detection heads, each trained for multiple areas, the base visual model pre-trained with a large number of images of industrial components from multiple areas can be efficiently utilized. Therefore, according to the prior art, it is not necessary to train a specific deep learning-based AOI detection model for each area.

[0034] Fig. 4 shows a block diagram of an AOI device for industrial components according to an embodiment.

[0035] The same reference numerals in Fig. 4 as in Fig. 3 denote the same units, and the corresponding parts are not described in detail. As in Fig. 4, the task detection models 320A to 320D include the classification models 3210A to 3210D, the segmentation models 3220A to 3220D, and the identification models 3230A to 3230D, respectively. For example, in the task detection model 320A, after the base visual model 310 receives the image I1A and generates the feature representation FM-A of the image I1A, the classification model 3210A in the task detection model 320A executes the classification task based on the feature representation FM-A and outputs the classification result 3310A, the segmentation model 3220A executes a segmentation task based on the feature representation FM-A and outputs a segmentation result 3320A, the identification model 3230A executes an identification task based on the feature representation FM-A and outputs an identification result 3330A.Similarly, after the base visual model 310 receives the image I1B and generates a feature representation FM-B of the image I1B, the classification model 3210B, the segmentation model 3220B, and the identification model 3230B in the task detection model 320B perform the classification task, the segmentation task, and the identification task, respectively, based on the feature representation FM-B to determine whether the component corresponding to the image is a faulty component.The classification models 3210C and 3210D, the segmentation models 3220C and 3220D, and the identification models 3230C and 3230D in the task detection models 320C and 320D also perform corresponding classification, segmentation, and identification tasks based on the feature representations FM-C and FM-D of the images I1C and I1D generated by the base visual model 310 to determine whether the component corresponding to the image is a defective component. Those skilled in the art will understand that the task detection models 320A to 320D are configured in the manner shown in FIG. Fig. 4, all include a classification model, a segmentation model, and an identification model, but in other embodiments, one or more of the task detection models 320A to 320D need not include all of the classification models, segmentation models, and identification models, but may include only one or more of the classification models, segmentation models, and identification models, and may also include other models in addition to the classification models, segmentation models, and identification models. Those skilled in the art will understand that the AOI detection module 1210 includes a base visual model 310 and a plurality of region-specific task detection models 320, while in other embodiments, the AOI detection module 1210 may include a base visual model 310 and a region-specific task detection model 320, as shown in Fig. 3 and Fig. 4 shown.

[0036] Fig. 5 shows a block diagram of an AOI device for industrial components according to an embodiment.

[0037] The same reference numerals in Fig. 5 as in Fig. 1, Fig. 3 and Fig. 4 denote the same units, and the corresponding parts are not described in detail. As in Fig. 5, each of the plurality of front ends 110A to 110D has a similar structure to that shown in Fig. 1, for example, the front ends 110A to 110D are each arranged in several different production lines, which cause the imaging devices 1110A to 1110D to take pictures of industrial components on the corresponding production lines, such as the ones shown in Fig. 3 and Fig. 4, and the captured images are provided to the processing system 120. The AOI detection module 1210 of the processing system 120 includes the basic visual model 310 and the task detection models 320A to 320D described above in connection with Fig. 3 and Fig. 4. The visual base model 310 processes the images I1A to I1D from the multiple areas of the front ends 110A to 110D to obtain feature representations FM-A to FM-D of the images. The feature representations FM-A to FM-D are input to the task detection models 320A to 320D, respectively, and the detection results 3310A to 3330A, 3310B to 3330B, 3310C to 3330C, and 3310D to 3330D are obtained from the task detection models 320A to 320D, respectively, based on the feature representations FM-A to FM-D. In the Fig. In the embodiment of the AOI system shown in Fig. 5, by integrating the visual base model and a plurality of task acquisition heads each associated with a plurality of areas for performing AOI acquisition tasks for a plurality of production lines, the AOI acquisition processing of a plurality of production lines can be processed in a pipeline by the integrated AOI acquisition model, thereby further increasing the utilization rate of the computing resources of the computing device 120.

[0038] It will be understood by those skilled in the art that the AOI detection module 1210, although described in Fig. 5 comprises a base visual model 310 and a plurality of domain-specific task capture models 320A to 320D, in other embodiments may also include a base visual model 310 and a task capture model 320 used to process the AOI capture of industrial components from a domain (e.g., a production line).

[0039] Fig. 6 shows a schematic representation of a contrastive learning method for training a visual base model according to an embodiment.

[0040] To train the visual base model 310, a large dataset with images of industrial components from various sectors must be created. Fig. In the example of the AOI system 500 shown in Figure 5 for a large company with production lines for many types of industrial components, the imaging units 1110 deployed on each production line can obtain images of multiple types of industrial components as training data sets for pre-training the baseline visual model. For example, in one embodiment, millions of images of industrial components from multiple production lines can be collected as an AOI training data set for pre-training a baseline visual model.

[0041] In one embodiment, the Fig. 6 can be applied to train the visual base model 310 to learn to extract feature representations from industrial component images in multiple areas. Using the example of the Fig. For the industrial component image I1 shown in Figure 6, the data enrichment module 610 performs enrichment processing on the image I1 to obtain an image pair (I1-1, I1-2). For example, the data enrichment module 610 may obtain an image pair (I1-1, I1-2) by performing a color change, a brightness change, random cropping, mirroring, rotation, etc. on the image I1, whereby the image pair may be referred to as a positive pair.

[0042] The base visual model 310 encodes the images I1-1 and I1-2 to obtain the feature representations FM-1 and FM-2 of the corresponding images, respectively. The comparison module 620 determines a similarity between the two images I1-1 and I1-2 based on the feature representations FM-1 and FM-2, which may, for example, be a cosine similarity between the feature vectors corresponding to the two images I1-1 and I1-2. In one example, the comparison module 620 may calculate a cosine similarity between the feature representations FM-1 and FM-2 as a similarity S1 between the two images I1-1 and I1-2.In another example, the comparison module 620 may further include additional neural network layers for converting FM-1 and FM-2 into feature representations FM-1Z and FM-2Z in a different space, and furthermore, the comparison module 620 calculates the cosine similarity between the feature representations FM-1Z and FM-2Z as the similarity S1 between the two images I1-1 and I1-2.

[0043] The loss determination module 630 can determine the loss value L1 based on the similarity S1 between the two images 11-1 and 11-2. For example, the loss value L1 can be determined based on the similarity between the positive pair (I1-1, I1-2) and the similarity between the augmented image and other images of a stack that form a negative pair. Determining the loss value based on the similarity of positive and negative pairs is a commonly used technical tool in the field of image processing and will therefore not be discussed in detail. The base visual model 310 can then be updated based on the comparative learning loss value L1. For example, a well-known AdamW optimizer can be used to update the base visual model 310 based on the loss L1.

[0044] Fig. 7 shows a schematic representation of a masked autoencoder (MAE) learning method for training a base visual model according to an embodiment.

[0045] In one embodiment, the Fig. 7 are applied to train the visual base model 310 to learn to extract feature representations from industrial component images in multiple areas. Using the example of the Fig. For the industrial component image I1 shown in Figure 7, the masking module 710 performs masking processing on the image I1 to obtain a masked image I1-M. For example, the masking module 710 divides the image I1 into small blocks, then randomly selects some of these small blocks, retains them, and masks all the remaining small blocks to obtain the masked image I1-M.

[0046] The visual base model 310 encodes the masked image I1-M to obtain the feature representation FM-M of the masked image. The decoder 720 decodes the restored image I1-R based on the feature representation FM-M. The loss determination module 730 determines the MAE loss value L2 based on the restored image I1-R and the original image I1. For example, the loss determination module 730 may use a mean square error (MSE) function as the loss function to determine the MAE loss value L2, that is, the original pixels in the original image I1 and the reconstructed pixels in the restored image I1-R are subtracted and then summed by squaring. The base visual model 310 may then be updated based on the MAE loss value L2, for example, a known AdamW optimizer may be used to update the base visual model 310 based on the loss L2.

[0047] In one embodiment, both the contrastive learning method and the MAE method described in Fig. 6 and Fig. 7 may be used to train the base visual model 310, e.g., a weighted sum of the loss values ​​L1 and L2 may be used as the loss value to update the base visual module.

[0048] Those skilled in the art will appreciate that any suitable training method can be used to train the base visual model 310. The base visual model 310, trained on a large dataset of industrial component images from multiple domains, learns both domain-specific feature representations and universal feature representations covering different domains. Therefore, the feature representation of the industrial component image extracted by the base visual model 310 as a feature extractor can help the downstream domain-specific AOI detection modules 320 learn AOI detection tasks adapted to different domains through a lightweight domain-specific adaptive training process.

[0049] Fig. 8 shows a schematic diagram for training a task detection model 320 according to an embodiment.

[0050] Using the example of Fig. 3 and Fig. 4, the task detection model 320 may include one or more classification models 3210, a segmentation model 3220, and an identification model 3230. In one embodiment, the classification model 3210 may be a multilayer perceptron (MLP) model, the segmentation model 3220 may be a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism, and the identification model 3230 may be a bounding box decoder with classification results or a decoder based on an attention mechanism of the CNN model.The task detection model 320 may be trained based on the image set collected and labeled in area D3 as a training dataset; more specifically, the classification model 3210, the segmentation model 3220, and / or the identification model 3230 may be trained therein. In one embodiment, the lightweight task heads 3210, 3220, and 3230 may be trained based on a small set of training datasets specific to area D3. The small set of training datasets may, for example, be a dataset in units of hundreds of images.

[0051] The base visual model 310 encodes the image I1 to obtain the feature representation FM of the image. The task detection model 320 obtains a prediction P about the detection task based on the feature representation. For example, the classification model 3210, the segmentation model 3220, and / or the identification model 3230 each obtain the predictions P1, P2, and / or P3 about the detection task based on the feature representation. The loss determination module 830 determines the loss value L3 based on the prediction results P1, P2, and / or P3 and the corresponding labels representing the labeled detection results, specifically including the loss values ​​L3-1, L3-2, and / or L3-3 corresponding to the prediction results P1, P2, and / or P3, respectively.The loss value L3 can be determined using various suitable methods for determining the supervision loss using the prediction value and the labeling value. The task detection model 320 can then be updated based on the supervision loss value L3. In particular, the classification model 3210, the segmentation model 3220, and / or the identification model 3230 can be updated based on the loss values ​​L3-1, L3-2, and / or L3-3. For example, the already known AdamW optimizer can be used to update the task detection model 320 based on the loss L3. In one embodiment, the parameters of the pre-trained visual base model 310 are frozen during the update of the task detection model 320.In another embodiment, a small number of parameters of the pre-trained base visual model 310 may be updated simultaneously during the update of the task capture model 320.

[0052] In the Fig. 8, the task detection model 320 may be trained with images of industrial components from different domains than the images of industrial components from multiple domains that were used to train the base visual model 310. For simplicity, the images shown in Fig. 3 represent multiple regions used to train the base visual model 310, and region D5 represents the region used to train the task detection model 320. Those skilled in the art will understand that the number of regions used to train the base visual model 310 is actually higher. The base visual model 310, trained on a large dataset of industrial component images from multiple regions D1 to D4, not only learns region-specific and general feature representations covering different regions, but the general feature representations can also be applied to regions unknown to the base visual model 310. Therefore, in this embodiment, a task detection model 320 can be trained that corresponds to region D5 not seen by the base visual model 310.This embodiment is particularly advantageous for performing the AOI of produced parts in the initial phase of setting up a new production line, which corresponds, for example, to the area D5.

[0053] Fig. 9 shows a flowchart of a method for performing an AOI according to an embodiment.

[0054] In step 910, an image of a captured component of a first region is obtained, wherein the first region corresponds to a first industrial component type or a first industrial component product line.

[0055] In step 920, the image of the detected component is input into a detection model comprising a submodel for universal feature extraction and at least one submodel for task detection related to a specific AOI task of the first region.

[0056] In step 930, a first feature representation of the image is generated by the universal feature extraction submodel based on the image of the detected component.

[0057] In step 940, at least one detection result relating to a specific AOI task of the first region is generated by the at least one sub-model for task detection based on the first feature representation of the image.

[0058] In one embodiment, the specific AOI task of the first region comprises at least one or at least two of the following tasks: a classification task, wherein the image of the detected component is to be classified into one of a plurality of categories, wherein the plurality of categories include, for example, normal, defective, or other; a segmentation task for segmenting the defective region in the image of the detected component, wherein the segmented target region is represented, for example, by a mask, wherein the pixel value of the target region is 1 and the pixel value of the remaining region is 0; and an identification task for identifying a defective region in the image of the detected component, wherein the identified region is represented, for example, by a bounding box.

[0059] In one embodiment, if the specific AOI task for this first region is a classification task, the submodel for task detection is an MLP model; if the specific AOI task of the first region is a segmentation task, the submodel for task detection is the mask decoder of the CNN model or the decoder based on the attention mechanism; and if the specific AOI task of the first region is an identification task, the submodel for task detection is a bounding box decoder with classification results of the CNN model or a decoder based on the attention mechanism.

[0060] In one embodiment, the submodel for universal feature extraction is a base visual model based on the attention mechanism.

[0061] In one embodiment, the universal feature extraction submodel is trained on the basis of images of industrial components in multiple regions, wherein the multiple regions each correspond to multiple industrial component types or multiple production lines for industrial components.

[0062] In one embodiment, the first region is a region within the plurality of regions. In one embodiment, the first region is a region outside the plurality of regions. The task detection submodel is trained on images of industrial components in the first region. In one embodiment, the image of the first region used to train the task detection submodel is different from, or not entirely identical to, the image of the first region used to train the universal feature extraction submodel.

[0063] In one embodiment, method 900 further comprises obtaining a second image of a second detected component of a second region, wherein the second region corresponds to a second industrial component type or a second industrial component production line; inputting the second image of the second detected component into the detection model, wherein the detection model further comprises at least one second task detection sub-model related to the specific AOI task of the second region; generating a first feature representation of this second image based on the second image of the second component detected by the universal feature extraction sub-model;at least one detection result related to the specific AOI task in the second region is generated by the at least one second sub-model for task detection based on the first feature representation of the second image;

[0064] Fig. 10 shows a flowchart of a method for training a detection model for AOI according to an embodiment.

[0065] In step 1010, images of industrial components in multiple areas are collected as a multi-area AOL training dataset, where the multiple areas correspond to multiple types of industrial components or multiple production lines for industrial components.

[0066] In step 1020, a universal feature extraction submodel is pre-trained in the detection model based on the training dataset of the multi-domain AOL to obtain a trained universal feature extraction submodel.

[0067] In step 1030, based on the images of industrial components in the first region as a training set of the single-region AOI, a task detection sub-model is trained that relates to a specific AOI task in the first region in the detection model, wherein the first region corresponds to a first industrial component type or a first industrial component production line, wherein the trained universal feature extraction sub-model receives the image in the training data set of the single-region AOI and outputs the first feature representation of the image, wherein the at least one task detection sub-model receives the first feature representation of the image and predicts at least one detection result that relates to a specific AOI task of the first region.

[0068] In one embodiment, in step 1020, the unlabeled multi-area AOL training dataset is data augmented to obtain a set of training image pairs of the multi-area AOL (pair), and the submodel for universal feature extraction is trained using a comparative learning method based on the set of training image pairs of the multi-area AOL; and / or the submodel for universal feature extraction is trained using the MAE method based on the unlabeled multi-area AOL training dataset.

[0069] In one embodiment, in step 1020, training the universal feature extraction submodel using the contrastive learning method on the set of training image pairs of the multi-region AOL comprises: receiving an image pair by the universal feature extraction submodel in the set of training image pairs of the multi-region AOL and outputting a first feature representation of an image pair, obtaining the similarity of the image pair by the contrastive learning module based on the first feature representation of the image pair, obtaining a contrastive learning loss based on the similarity of the image pair, and updating the universal feature extraction submodel based on the contrastive learning loss;and / or the use of the MAE method for training the universal feature extraction submodel comprises: randomly masking the original images in the multi-region AOL training dataset to obtain the masked images, wherein the universal feature extraction submodel receives the masked images and outputs a first feature representation of the masked image, wherein the decoder module predicts a restored image based on the first feature representation of the respective masked image, wherein the MAE loss is obtained based on the restored image and the original image, and the universal feature extraction submodel is updated based on the MAE loss;

[0070] In one embodiment, in step 1030, the submodel for task detection is trained based on the labeled training data set of the single-domain AOL.

[0071] In one embodiment, the method 1000 further comprises: at least one second task detection sub-model related to the specific AOI task in the second region and trained based on industrial component images as a second single-region AOI training dataset, wherein the second region corresponds to the second industrial component type or the second industrial component production line, wherein the trained universal feature extraction sub-model receives the second image in the second single-region AOI training dataset and outputs the first feature representation of the second image, wherein the at least one second task detection sub-model receives the first feature representation of the second image and predicts at least one detection result related to a second specific AOI task of the second region.

[0072] Fig. 11 shows a block diagram of an apparatus for AOI of industrial components according to an embodiment.The device 1100 comprises: an image acquisition module 1110 used for acquiring an image of a detected component of a first region, wherein the first region corresponds to a first industrial component type or a first production line for industrial components; a detection module 1120, wherein the detection model comprises a universal sub-model for feature extraction and at least one sub-model for task detection associated with a specific AOI task for this first region, wherein a first feature representation of the image is generated from the image of the detected component by means of the universal sub-model for feature extraction, wherein the sub-module for the at least one task detection generates at least one detection result related to a specific AOI task of the first region based on the first feature representation of the image.

[0073] In one embodiment, the image acquisition module 1110 acquires a second image of a second detected component of a second region, wherein the second region corresponds to a second industrial component type or a second industrial component production line, wherein the acquisition module 1120 further comprises at least a second sub-module for task acquisition related to the specific AOI task of the second region, wherein the sub-module for universal feature extraction generates a first feature representation of the second image based on the second image of the second detected component, wherein the sub-module for the at least one second task acquisition generates at least one first acquisition result related to a specific AOI task of the second region based on the second feature representation of the image.

[0074] In one embodiment, the Fig. 11 shown detection module 1120 which is described in the above text in connection with Fig. 1, Fig. 3, Fig. 4 and Fig. 5 described acquisition model 1210, wherein the sub-module for universal feature extraction and the sub-module for task acquisition each comprise the model described in the above text in conjunction with the model described in Fig. 3, Fig. 4 and Fig. 5 described visual base model 310 and the task capture model 320.

[0075] Fig. Figure 12 shows a block diagram of a processing system for AOI of industrial components according to one embodiment. The control system or processing system 1200 may include one or more control units or processing units 1210, wherein the control unit 1210 executes one or more machine-readable instructions stored or encoded in a machine-readable storage medium (e.g., memory 1220). Although in Fig. 12, it will be understood by those skilled in the art that the control system 1200 may include a variety of other components, such as various communication modules, bus modules, and possibly user interface modules. In one embodiment, the processing unit is configured to, when executing the program instructions, perform the various steps described above in connection with Fig. 1 to 11.

[0076] According to one embodiment, a program product, such as a non-transitory machine-readable medium, is provided. The non-transitory machine-readable medium may contain instructions that, when executed by the processing unit 1210, may perform the various operations and functions described above in connection with the Fig. 1 to 11 in various embodiments of the present application.

[0077] The specific embodiments set forth above in conjunction with the figures describe exemplary embodiments and do not represent all embodiments that may be practiced or fall within the scope of the claims. The term "exemplary," as used throughout the specification, means "serving as an example or illustration" and does not imply "preferred" or "advantageous" over other embodiments. To provide an understanding of the described technologies, the specific embodiments include specific details. However, these technologies may be implemented without these specific details. In some cases, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0078] The foregoing description of the present application enables those of ordinary skill in the art to practice or apply the contents of the present application. Various modifications of the present application will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of the present application. Therefore, the present application is not limited to the examples and embodiments described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.

Claims

[1] Automatic Optical Inspection (AOI) method comprising: Obtaining an image of a captured component of a first region, the first region corresponding to a first industrial component type or a first industrial component production line; Inputting the image of the detected component into a detection model, the detection model comprising at least one submodel for universal feature extraction and at least one submodel for task detection related to a specific AOI task of the first region; wherein the universal feature extraction submodel generates the first feature representation of the image based on the image of the detected component; wherein the at least one sub-model for task detection generates at least one detection result based on the first feature representation of the image, which detection result relates to the specific AOI task of the first region. [2] The method of claim 1, wherein the AOI task of the first region comprises at least one or at least two of the following tasks: Classification task to classify the image of the captured component into one of several categories; Segmentation task, where the defective section in the image of the detected component is segmented; Identification task, where the defective section is identified in the image of the captured component. [3] The method of claim 2, wherein, when the specific AOI task of the first region is the classification task, the submodel for task detection is a multi-layer perceptron (MLP) model; where, if the specific AOI task of the first region is the segmentation task, the submodel for task detection is a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism; and where, if the specific AOI task of the first region is the identification task, the submodel for task detection is a bounding box decoder with classification results of the CNN model or a decoder based on an attention mechanism. [4] The method according to any one of claims 1 to 3, wherein the submodel for universal feature extraction is a visual base model based on an attention mechanism. [5] The method of claim 4, wherein the universal feature extraction sub-model is trained based on images of industrial components in a plurality of regions, each of the plurality of regions corresponding to a plurality of industrial component types or a plurality of industrial component production lines. [6] The method of claim 5, wherein the first region is a region within a plurality of regions or the first region is a region outside the plurality of regions, wherein the task detection sub-model is trained using the images of the industrial components in the first region. [7] The method of claim 1, further comprising: Obtaining a second image of a second detected component of a second region, wherein the second region corresponds to a second industrial component type or a second production line for industrial components; inputting the second image of the second detected component into the detection model, the detection model further comprising at least a second sub-model for task detection related to the specific AOI task of the second region; wherein the general feature extraction submodel generates the first feature representation of the second image based on the second image of the second detected component; wherein the at least second sub-model for task detection generates at least one detection result based on the first feature representation of the second image, which detection result relates to the specific AOI task of the second area. [8] A method for training a detection model for Automatic Optical Inspection (AOI), comprising: Collecting images of industrial components as a multi-region AOI training dataset, wherein the multiple regions each correspond to multiple industrial component types or multiple industrial component production lines; where the submodel for universal feature extraction in the detection model is trained on the basis of the multi-domain AOI training dataset, to obtain a trained sub-module for universal feature extraction; wherein, based on images of industrial components in the first area as a single-area AOI training dataset, at least one sub-model for task detection is trained, which sub-model relates to a specific AOI task in the first area in the detection model, wherein the first area corresponds to the first industrial component type or the production line of the first industrial component, wherein the sub-model for universal feature extraction receives an image from the single-area AOI training dataset of the first area and outputs a first feature representation of the image, wherein the at least one sub-model for task detection receives the first feature representation of the image and predicts at least one detection result related to a specific AOI task in the first area. [9] The method of claim 8, wherein training the universal feature extraction submodel based on the multi-domain AOL training dataset comprises: Performing data augmentation on the unlabeled multi-area AOI training dataset to obtain a set of training image pairs of the multi-area AOI, and training the submodel for universal feature extraction using a contrastive learning method based on the set of training image pairs of the multi-area AOI; and / or Training the universal feature extraction submodel using the masked autoencoder (MAE) method on the unlabeled multi-domain AOL training dataset. [10] Method according to claim 9, wherein training the universal feature extraction submodel using the contrastive learning method on the training set of the multi-domain AOL comprises: receiving an image pair by the universal feature extraction submodel in the set of the training image pair of the multi-domain AOL and outputting a first feature representation of an image pair, obtaining the similarity of the image pair by the contrastive learning module based on the first feature representation of the image pair, obtaining a contrastive learning loss based on the similarity of the image pair, and updating the universal feature extraction submodel based on the contrastive learning loss; and / or wherein the method for training the submodel for universal feature extraction using the MAE method comprises: masking the original image in the training dataset of the multi-region AOL to obtain a masked image, obtaining the masked image by the submodel for universal feature extraction and outputting a first feature representation of the masked image, predicting a restored image by a decoder module based on a first feature representation of the masked image, obtaining an MAE loss based on the restored image and the original image, and updating the submodel for universal feature extraction based on the MAE loss. [11] The method of claim 8, wherein training at least one task detection sub-model in the detection model related to a specific AOI task in the first region based on industrial component images in the first region as a single-region AOI training dataset for a single region comprises: training the task detection sub-model on the labeled single-region AOI training dataset. [12] The method of claim 8, further comprising: Based on industrial component images in the second region as a second single-region AOI training dataset, at least one second sub-model for task detection is trained, which sub-model relates to a specific second AOI task in the second region in the detection model, wherein the second region corresponds to the second industrial component type or the production line of the second industrial component, wherein the sub-model for universal feature extraction receives a second image from the second single-region AOI training dataset and outputs a first feature representation of the second image, wherein the at least one second sub-model for task detection receives the first feature representation of the image and predicts at least one detection result related to a second specific AOI task in the second region. [13] Automatic Optical Inspection (AOI) device comprising: an image acquisition module for obtaining an image of the detected part in a first area, the first area corresponding to a first type of industrial component or corresponds to a first production line for industrial components; a detection module, wherein the detection model comprises a sub-module for universal feature extraction and at least one sub-module for task detection related to the specific AOI task of the first region, wherein the sub-module for universal feature extraction generates a first feature representation of the image based on the component detected on the image, wherein the at least one sub-module for task detection generates at least one detection result related to a specific AOI task of the first region based on the first feature representation of the image. [14] The apparatus of claim 13, wherein the image acquisition module obtains a second image of a second detected component in a second region, the second region corresponding to a second industrial component type or a second industrial component production line, the detection module further comprising at least one second task detection sub-module related to the specific AOI task of the second region, the universal feature extraction sub-module generating a first feature representation of the second image based on the second component detected on the second image, the at least one second task detection sub-module generating at least one detection result related to a specific AOI task of the second region based on the first feature representation of the second image. [15] Automatic Optical Inspection (AOI) system comprising: an image capture device for capturing images of the detected components; one or more processors; and one or more memories storing computer-executable instructions that, when executed by the one or more processors, perform operations according to any one of claims 1 to 12. [16] Computer system comprising: one or more processors; and one or more memories storing computer-executable instructions that, when executed by the one or more processors, perform operations according to any one of claims 1 to 12. [17] A machine-readable storage medium storing executable instructions which, when executed, cause one or more processors to perform a method according to any one of claims 1 to 12.

Images