Defect detection method and device, storage medium and electronic device

Abstract

Provided are a defect detection method and device, a computer readable storage medium, and an electronic device. The method comprises: obtaining a detection task and a plurality of types of images corresponding to the detection task; obtaining a plurality of defect detection models corresponding to the types of images, the defect detection models being trained by the same initial model; and performing defect detection on the images of the types by using the defect detection models corresponding to the types of the images to obtain defect detection results. The technical solution improves detection efficiency and reduces development costs.

Description

Technical Field

[0001] The present disclosure relates to the field of defect detection technology, and in particular to a defect detection method and device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In the field of production and processing technology, problems with equipment, parameters, operations, environmental interference and other aspects may cause defective products. With the rise of artificial intelligence algorithms represented by deep learning, the use of deep algorithm learning models for defect detection is becoming more and more widespread.

[0003] However, in the existing technology, deep algorithm learning models often require manual screening of normal images, and the same algorithm cannot be used to detect different types of images, which wastes human resources and requires the development of different models to realize defect detection for different types of images. The detection efficiency is low and the development cost is high.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The purpose of the present disclosure is to provide a defect detection method, a defect detection device, a computer-readable medium, and an electronic device, thereby improving detection efficiency and reducing development costs, at least to a certain extent.

[0006] According to a first aspect of the present disclosure, there is provided a defect detection method, comprising:

[0007] Acquire a detection task and acquire multiple types of images corresponding to the detection task;

[0008] Obtaining defect detection models corresponding to the types of the images, respectively, obtained by training with the same initial model;

[0009] Defect detection is performed on various types of images using the defect detection model corresponding to the type of each image to obtain defect detection results.

[0010] According to a second aspect of the present disclosure, there is provided a defect detection device, comprising:

[0011] An image acquisition module is used to acquire a detection task and obtain multiple types of images corresponding to the detection task;

[0012] A model acquisition module, configured to acquire defect detection models corresponding to the types of the images and obtained by training with the same initial model;

[0013] The defect detection module is used to perform defect detection on various types of images using the defect detection model corresponding to the type of each image to obtain defect detection results.

[0014] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0015] According to a fourth aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] processor; and

[0017] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0020] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0021] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure may be applied;

[0022] Figure 3 A flow chart schematically illustrates a defect detection method in an exemplary embodiment of the present disclosure;

[0023] Figure 4 A flowchart schematically illustrates an algorithm of a defect detection method in an exemplary embodiment of the present disclosure;

[0024] Figure 5 Schematically illustrates an architecture diagram of a defect strategy model in an exemplary embodiment of the present disclosure;

[0025] Figure 6 Schematically illustrates an operator interface diagram in an exemplary embodiment of the present disclosure;

[0026] Figure 7The figure schematically shows the composition of the defect detection method and device in the exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0029] Figure 1 A schematic diagram shows a system architecture of an exemplary application environment in which a defect detection method and apparatus according to an embodiment of the present disclosure can be applied.

[0030] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, and tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as needed. For example, the server 105 may be a server cluster consisting of multiple servers.

[0031] The defect detection method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the defect detection device is generally set in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the defect detection method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the defect detection device can also be set in the server 105, and this is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, a user can obtain multiple types of images corresponding to the product based on the product information through the terminal devices 101, 102, and 103 and upload them to the server 105. The server obtains the defect detection results through the defect detection method provided in the embodiment of the present disclosure and transmits the defect detection results to the terminal devices 101, 102, and 103, etc.

[0032] An exemplary embodiment of the present disclosure provides an electronic device for implementing a defect detection method, which may be Figure 1 The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, wherein the memory is used to store executable instructions of the processor, and the processor is configured to perform the defect detection method by executing the executable instructions.

[0033] Below Figure 2 The structure of the electronic device is exemplarily described by taking the mobile terminal 200 in FIG. 1 as an example. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in FIG. 2 can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0034] like Figure 2As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, and the like.

[0035] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0036] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in the mobile terminal 200, such as image recognition, face recognition, speech recognition, and text comprehension.

[0037] The processor 210 is provided with a memory that can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution of the instructions is controlled by the processor 210.

[0038] The charging management module 240 is configured to receive charging input from a charger. The power management module 241 is configured to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 290, the camera module 291, the wireless communication module 260, and the like.

[0039] The wireless communication function of the mobile terminal 200 can be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, and the baseband processor, and the like. The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. The mobile communication module 250 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the mobile terminal 200. The modem processor can include a modulator and a demodulator. The wireless communication module 260 can provide a solution for wireless communication including a wireless local area network (WLAN) such as a wireless fidelity (Wi-Fi) network, Bluetooth (BT), and the like applied to the mobile terminal 200. In some embodiments, the antenna 1 of the mobile terminal 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the mobile terminal 200 can communicate with a network and other devices through wireless communication technology.

[0040] The mobile terminal 200 can implement a display function through a GPU, a display screen 290, and an application processor, and the like. The GPU is a microprocessor for image processing, which is connected to the display screen 290 and the application processor. The GPU is configured to perform mathematical and geometric calculations for graphics rendering. The processor 210 can include one or more GPUs that execute program instructions to generate or change display information.

[0041] The mobile terminal 200 can implement a photographing function through an ISP, a camera module 291, a video codec, a GPU, a display screen 290, and an application processor, and the like. The ISP is configured to process data fed back by the camera module 291. The camera module 291 is configured to capture a still image or a video. The digital signal processor is configured to process digital signals, which can include processing digital image signals and processing other digital signals. The video codec is configured to compress or decompress digital video. The mobile terminal 200 can support one or more video codecs.

[0042] The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 via the external memory interface 222 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0043] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.

[0044] The mobile terminal 200 can implement audio functions such as music playback and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the headphone jack 274 and the application processor.

[0045] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be set in the camera module 291.

[0046] Pressure sensor 2802 is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 2802 can be provided on display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0047] The gyroscope sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In some embodiments, the angular velocity of the mobile terminal 200 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 2803. The gyroscope sensor 2803 can be used for image stabilization, navigation, somatosensory gaming scenarios, etc.

[0048] In addition, sensors with other functions can be set in the sensor module 280 according to actual needs, such as air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, bone conduction sensor, etc.

[0049] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, the keys 294 include a power key, a volume key, etc., through which the user can input key signals related to user settings and function control of the mobile terminal 200. Another example includes the indicator 292, the motor 293, and the SIM card interface 295.

[0050] In related technologies, a product is photographed and inspected from different dimensions (overall / local, high / low resolution, grayscale / color images). During the screen production process, problems with equipment, parameters, operations, environmental interference, and other links may cause defects in the output products. After each process, optical (AOI (Automated Optical Inspection)) inspection is used, and many different types of image data (AOI (Automated Optical Inspection) color images, TDI (Time Delayed and Integration) images, and DM (digital micro) images) are generated. Professional operators are required to grade these images. With the rise of artificial intelligence algorithms represented by deep learning, AI (Artificial Intelligence) algorithms are introduced into the process of grading defective images, and systems for automatically detecting defective images (ADC (Automatic defect detection and classification system)) have emerged.

[0051] During the screen production process, in the quality inspection link, optical AOI equipment will be used to take pictures of the circuits on the glass substrate. First, the industrial camera CCD will take a picture of each glass substrate (covered with circuits) and generate a grayscale image of the entire glass plate, which is the DM image. Each glass substrate will only generate one DM image. Figure 1 Generally, it is a single-channel grayscale image of 1500x1500 or more. If there is a defective or suspicious point, the TDI image and AOI color image of the point will be captured using a microscope. Figure 1Generally, the single-channel 64x64 grayscale image is a "blurred image" of the defect point. Since the image is small, it is easy to generate and transmit, and about 500 TDI images are generated for each glass substrate. The AOI color image Figure 1 Generally, the three-channel 1360x1020 RGB image is a high-definition display image of the defect point. Since it is not easy to generate and transmit, about 150 AOI color images are generally captured for each glass substrate.

[0052] In the production process, after the above three types of images of each glass substrate are captured by the AOI device, a person determines whether each image has a real defect and gives the specific category of the defect by distinguishing the above three types of images. In the production of a factory, millions of images are generated every day, and a large amount of manpower is needed for defect detection. Moreover, due to the limited energy of a person, misjudgment and omission often occur, which ultimately affects the yield of the product.

[0053] The defect detection method and the defect detection device of the exemplary embodiment of the present disclosure are described in detail below.

[0054] Figure 3 The flow of a defect detection method in the present exemplary embodiment is shown, which includes the following steps:

[0055] In step S310, a detection task is obtained, and a plurality of types of images corresponding to the detection task are obtained.

[0056] In step S320, a defect detection model trained by the same initial model corresponding to the type of each image is obtained.

[0057] In step S330, the defect detection model corresponding to the type of each image is used to detect defects in each type of image to obtain a defect detection result.

[0058] Compared with the prior art, the present disclosure uses a defect detection model to complete the detection of product defects, uses a plurality of types of images of the product, and uses different defect detection models for different types of images. Since the different defect detection models of the present disclosure are all trained by the same initial model, the same algorithm can be used to obtain the defect detection model, which saves development resources and reduces the cost of defect detection.

[0059] In step S310, a detection task is obtained, and a plurality of types of images corresponding to the detection task are obtained.

[0060] In an example embodiment of the present disclosure, an inspection task may be first obtained, and then product information corresponding to the inspection task may be obtained based on the inspection task. Furthermore, multiple types of images of the product may be obtained based on the product information. The different types of images may be represented by different resolutions, different numbers of channels, or both different resolutions and different numbers of channels.

[0061] In this example embodiment, the product information includes information such as the product name and product site. This information can also be customized based on user needs, which is not specifically limited in this example embodiment. The various types of images can be obtained by photographing the product using cameras with different configuration parameters. Specifically, the various types of images can include product-specific DM images, TDI images, AOI color images, etc., which are not specifically limited in this example embodiment.

[0062] In one exemplary embodiment of the present disclosure, the product information can be obtained based on the product information field. The server receives the inspection task sent by the training system and parses the task. To determine the product name / site name from the inspection task, the inspection task includes the product information field. By parsing the field, the server can clearly determine the site name and product name corresponding to the inspection task.

[0063] In this example embodiment, specifically, to obtain multiple types of images corresponding to the product according to the above product information, path information corresponding to various types of images in the product information can be first obtained, and then multiple types of images corresponding to the product can be obtained in the repository according to the above path information.

[0064] In this example embodiment, after acquiring different types of images corresponding to the inspection task, the defect detection method of the present disclosure may further include preprocessing the acquired different types of images. Specifically, the number of channels of each type of image may be set to the same. The maximum number of channels in each type of image may be determined, and the number of channels of all images may be set to the maximum number of channels. The size of each type of image may then be adjusted to a preset size corresponding to the respective type.

[0065] For example, various types of images including DM images, TDI images, and AOI color images are explained in detail. Among them, DM images and TDI images are single-channel images, while AOI color images are three-channel RGB images. In order to be compatible with the above three images, the DM images and TDI images are subjected to channel expansion processing in the preprocessing operation, that is, the single channel is copied three times and converted into a three-channel image for processing. Experimental verification shows that this processing method does not affect the accuracy of the algorithm.

[0066] Furthermore, a resolution threshold may be set according to the defect detection model. The threshold may be determined by the hardware system in which the model is located, or may be customized according to user needs, and is not specifically limited in this exemplary embodiment.

[0067] For example, the above-mentioned resolution threshold can be 1000 to 1200 with the shortest side, such as 1020, 1036, etc., which is not specifically limited in this example embodiment. Each type of image will be scaled to a different preset resolution and processed into a three-channel image for input into the defect detection model. For example, the DM image will be scaled to a three-channel image with a shortest side of 1200 and input into the defect detection model. Since the TDI image has a lower resolution, over-magnification will cause image distortion. Therefore, the TDI image can be scaled to a three-channel image of 256x256 and input into the defect detection model. The AOI color image can be changed, for example, maintaining 1360x1020, or it can be placed in a three-channel image with a shortest side greater than or equal to 1000 and input into the defect detection model. At the same time, each type of image will be normalized accordingly based on the mean and variance of each data set. The preset resolution corresponding to different types of images can also be customized according to user needs, which is not specifically limited in this example embodiment.

[0068] In this example embodiment, a preprocessing parameter modification interface may be provided for the preprocessing operation, so that the user can adjust the parameter information of the preprocessing operation through the preprocessing parameter modification interface. The preprocessing parameter modification interface may include a resolution modification interface and a channel number modification interface. Other modification interfaces may be added based on user needs. This is not specifically limited in this example embodiment. The user can modify the above parameters such as resolution and channel number through the above preprocessing parameter modification interface.

[0069] In another exemplary embodiment of the present disclosure, the server may first obtain information about multiple products corresponding to an inspection task, and then obtain images corresponding to each piece of product information with the same resolution and number of channels. Specifically, if the inspection task includes defect detection tasks for different products, the different types of images obtained may be images of different products or sites, but with the same resolution and number of channels. Specifically, the images may be images of the same format, but with at least one of the site and product being different.

[0070] In step S320, defect detection models corresponding to the types of the images and trained by the same initial model are obtained.

[0071] In an exemplary embodiment of the present disclosure, a model acquisition path is obtained according to the product and site, and then a defect detection model corresponding to the image type is obtained from a repository according to the image type.

[0072] In this example embodiment, the aforementioned multiple types of defect detection models can be trained using the same initial model. Specifically, an initial model is first obtained, and then training data corresponding to various types of images is obtained. The training data may include normal images and normal image information; defective images and image defect information. The server can then use the training data corresponding to each type of image to train the initial model, thereby obtaining defect detection models corresponding to the respective image types.

[0073] Before training, the parameters of the initial models corresponding to various image types are identical, eliminating the need to design separate initial models for each type of image, thus reducing design costs. The training process modifies model parameters using training data. Therefore, after training, the configuration parameters within the initial model are modified differently based on the training data, resulting in defect detection models corresponding to various image types.

[0074] For example, the DM diagram corresponds to DM model xx, the TDI diagram corresponds to TDI model xx, and the AOI color diagram corresponds to AOI model xx, where xx represents the model version information. The version is determined when the algorithm calls the system and passes in the model path message. That is, for each specific algorithm task, a set of three specific versions of model information will be passed in.

[0075] When performing defect detection, the commonly used method is the target detection algorithm. In defect detection scenarios, normal images often account for 60% or even more of all images. These normal images have no "targets" for the target detection algorithm. How to correctly classify normal images and defect images is a major problem in defect detection scenarios. The commonly used method is to first classify the images into normal images and defect images, and then input the defect images into the defect detection algorithm for defect category segmentation and positioning. The processing flow is complicated.

[0076] In defect detection scenarios, defects only occur in a small fraction of samples, and 50% or more of the input images are normal, defect-free. Traditional target detection algorithms require labeling the target category and location in each image. However, for normal images used in defect detection, the required "target" does not exist. Therefore, in this example implementation, the entire normal image is input into the network as a target. In addition to the defect category, a new category of normal images can be added, where the "defect" location is the entire image. This means that normal images and image normality information are included in the training data. This enables the defect detection model to classify normal and defective images, achieving accuracy rates exceeding 95% and recall rates exceeding 85% for normal images.

[0077] In the present example embodiment, the above-mentioned defect detection model is mainly a neural network model based on deep learning. For example, the defect detection model can be based on a feedforward neural network. The feedforward network can be implemented as a directed acyclic graph, with nodes arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within each layer. Data received at the nodes of the input layer of the feedforward network is propagated (i.e., “fed forward”) to the nodes of the output layer via an activation function that computes the state of the nodes of each successive layer in the network based on coefficients (“weights”) respectively associated with each of the edges connecting these layers. The output of the defect detection model can take various forms, which are not limited by the present disclosure. The defect detection model can also include other neural network models, for example, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a generative adversarial network (GAN) model, but is not limited thereto, and other neural network models known to those skilled in the art can also be employed.

[0078] The defect detection model generally needs to be obtained by training. The above-mentioned training of the initial model using a training algorithm can include the following steps: selecting a network topology; using a set of training data representing the problem modeled by the network; and adjusting the weights until the network model performs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to an input representing an instance in the training data set is compared to the “correct” labeled output for that instance; an error signal representing the difference between the output and the labeled output is computed; and when the error signal is propagated backwards through the layers of the network, the weights associated with the connections are adjusted to minimize the error. When the error for each output generated from an instance of the training data set is minimized, the initial model is considered to have been “trained” and defined as a defect detection model, and can be used for artificial intelligence inference tasks.

[0079] In step S330, defect detection is performed on the various types of images using the defect detection model corresponding to the type of each image to obtain a defect detection result.

[0080] In the present example embodiment, a defect influence weight can be set for each type of image according to a configuration parameter; defect detection is performed on the various types of images using the defect detection model corresponding to the type of each image to obtain a reference defect detection result corresponding to each type of image respectively; and the defect detection result is determined by fusing the defect influence weight and the reference defect detection result.

[0081] For example, the defect impact weights of the DM map, TDI map, and AOI color map can be set to 1:2:7, or 1:3:6. They can also be customized according to user needs and are not specifically limited in this example real-time mode.

[0082] In an example implementation, defect detection is performed on various types of images using defect detection models corresponding to the types of images to obtain defect detection results. After obtaining the defect detection results of various types of images, no fusion processing may be performed.

[0083] When performing defect detection on the above-mentioned DM images, TDI images, and AOI color images, different defect detection models trained with different neural network models can be used for separate detection to obtain defect categories, but there are differences in defect recognition accuracy and consistency.

[0084] The present disclosure trains different types of images of the same product separately using the same initial neural network model, and performs specific pre-processing and post-processing designs, thereby greatly improving the accuracy of defect detection. The pre-processing process has been described in detail above, and specifically may include setting the number of channels of the various types of images to be the same, scaling each type of image to a different preset resolution, etc. The post-processing operation may include setting defect impact weights for the various types of images according to configuration parameters; performing defect detection on the various types of images using the defect detection models corresponding to the types of images to obtain reference defect detection results corresponding to the various types of images; and determining the defect detection results by fusing the defect impact weights with the reference defect detection results.

[0085] It should be noted that the above pre-processing and post-processing have been described in detail above, so they will not be repeated here.

[0086] In this example embodiment, referring to Figure 5 As shown, the above-mentioned defect detection model can belong to a two-stage target detection algorithm, which can specifically include a feature extraction network and a defect recognition network. Among them, the feature extraction network can first be used to extract features from various types of images to obtain feature images; and the feature images can be standardized.

[0087] Specifically, the feature extraction network is used to extract features from the image to be inspected 510. It is a feature extraction layer 520, which is composed of basic deep learning units such as convolutional layers and pooling layers. The specific ones are similar to the classic VGG model, the ResNet model with a residual structure, and the lightweight MobileNet model, depending on the specific project requirements. In this example, a convolutional network composed of blocks with a residual structure is used to extract features. The remaining structures all expose interfaces and can be selected in the configuration file. The feature image obtained above can be sent to the candidate frame extraction network, and the candidate frames in the image are preliminarily screened into foreground frames (with defects) and background frames (without defects), and the coordinates of the candidate frames are adjusted by regression, that is, the feature image is screened to obtain the target feature image, and the defect category and coordinates of the target candidate feature image are determined; the defect detection result is obtained according to the preset screening strategy and the defect category and coordinates of the target feature image.

[0088] In this exemplary embodiment, the target feature image is normalized using the normalization layer 530 . ROI Pooling or ROI Align can be used, although this is not limited to image-based processing in this exemplary embodiment. The candidate boxes are normalized to a uniform size and input to the defect recognition network. In this example, ROI Align is used for normalization. In this exemplary embodiment, a normalization parameter adjustment interface can be configured to allow the user to adjust normalization parameters.

[0089] In this example embodiment, a defect recognition network is used to classify defects based on standardized feature images to obtain defect categories for each feature image, and to determine the coordinates of each feature image; and defect detection results are obtained based on the defect categories and coordinates.

[0090] Specifically, the standardized feature image can be input into a defect recognition network, which primarily consists of fully connected layers and softmax540. Convolutional layers are not required in the defect recognition network, and replacing them with convolutional layers significantly reduces the model size. Within the defect recognition network, defect categories can be classified and coordinates can be regressed.

[0091] Approximately 2,000 feature images with defect categories and coordinates were obtained. The candidate feature images were first filtered using the NMS algorithm, followed by a threshold adjustment strategy. Based on the results of multiple experiments for each image type, a specific threshold screening strategy was proposed. For each category in each image, an appropriate threshold was selected to filter the defect categories and obtain the target feature images. These thresholds can be customized based on user needs and are not specifically modified in this example implementation.

[0092] In this example implementation, the NMS algorithm is a general algorithm in face recognition and defect detection algorithms, which is mainly used to filter a large number of feature images processed by the algorithm according to the confidence scores and whether the coordinates overlap. The specific process is: the NMS algorithm receives thousands of feature image coordinates obtained by the previous algorithm and the confidence score of each feature image, first takes out the feature image with the highest confidence, calculates the intersection and union ratio with the remaining feature images in turn, deletes the feature images that exceed the threshold (exceeding the threshold means that the two feature images have a high degree of overlap, and the feature images are the same object), and then saves the feature image with the highest score. Next, take out the feature image with the highest confidence from the remaining ones, and repeat the previous step until the loop ends. In this way, non-overlapping feature images can be obtained on each image.

[0093] In this example embodiment, the number of target feature images obtained through screening may be 10, 20, etc., which is not specifically limited in this example embodiment.

[0094] In this example embodiment, defect detection results can also be obtained based on the defect categories and coordinates of the set screening strategy and target feature image. Specifically, among the multiple target feature images finally obtained, the preset screening strategy is used to determine and select the final defect category and coordinates based on the importance of the defect to the business and the frequency of the defect. Among them, the preset screening strategy includes: if the category appears in the final multiple target feature images, it is judged as this category; if two or more specific categories appear in the candidate box at the same time, one of the specific categories is selected; the largest category is selected according to the ranking of the target feature image; if all the target feature images are normal categories, the normal category is selected, and the "defect" coordinates are the original image size. The above preset screening strategy can also be customized according to user needs, that is, a customized result post-processing method to cope with a variety of image types and scenes, which is not specifically limited in this example embodiment.

[0095] In an example implementation of the present disclosure, a defect category and the coordinates of this category can be given for the DM map, TDI map, and AOI color map in each of the above tasks, and then the obtained results can be packaged and pushed to the salesperson's operating system for manual review.

[0096] In this example embodiment, referring to Figure 4As shown, the above defect detection method is introduced taking the AOI color map, TDI map and DM map as examples of multiple types of images. First, step S410 can be performed to input the AOI color map, TDI map and DM map, then step S420 is performed for preprocessing operation including AOI color map preprocessing, TDI map preprocessing and DM map preprocessing. The specific process of the preprocessing operation has been described in detail above, and thus will not be described here again. Then, step S430 can be performed to input the images to the defect detection model, and then steps S440 and S450 are performed for post-processing and obtaining the AOI color map defect detection result, TDI map defect detection result and DM map defect detection result. The post-processing includes AOI color map post-processing, TDI map post-processing and DM map post-processing, that is, the output of the model is post-processed to obtain the above defect detection result.

[0097] With reference to Figure 6 As shown, the maximum display area on the interface operated by the service personnel is the AOI color map 610, and the upper right corner display area 620 can switch the AOI color map to the AI recognized image and the AI unrecognized image, as well as the TDI map, through the switching identifier 640. Among them, the thumbnails of the AI recognized image and the TDI image can give the defect category below, and the operator only needs to roughly browse the thumbnails, which greatly increases the picture judging speed of the operator. The AI unrecognized image needs the operator to perform picture judgment in the middle display area, and gives the corresponding defect category of the picture. The DM map can be displayed in the lower right corner display area 630, and the defect is given above the picture, and the operator is provided with a rejudgment function.

[0098] In summary, the defect detection method in the present disclosure provides a high-efficiency and convenient processing method for the DM map, TDI map and AOI map of screen production detection based on specific business scenarios. Joint processing of multiple different types of images greatly improves the image detection efficiency and the subsequent manual work efficiency. At the same time, reasonable reuse of the same algorithm network and extraction of key parameters enhance the scalability of the algorithm. Finally, the classification of normal images and defect images and the defect subdivision and positioning of defect images are fused in the same end-to-end task process, which simplifies the task difficulty and improves the task processing speed.

[0099] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not for limiting purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0100] Further, with reference to Figure 7As shown, the embodiment of the present example further provides a defect detection apparatus 700, comprising an image acquisition module 710, a model acquisition module 720 and a defect detection module 730. Wherein:

[0101] The image acquisition module 710 can be configured to acquire a detection task and acquire a plurality of types of images corresponding to the detection task.

[0102] In an example embodiment, the image acquisition module 710 can extract a product information field in the detection task to acquire product information, acquire product information corresponding to the detection task; acquire a plurality of types of images according to the product information, specifically, acquire a plurality of types of images of a product corresponding to the product information; wherein the various types of images are obtained by photographing the product by a camera with different configuration parameters, and the various types of images include one or more of AOI color images, TDI images and DM images of the product.

[0103] The image acquisition module 710 can also perform preprocessing operations on each type of image to make the channel numbers of each type of image the same, specifically, the maximum channel number in each type of image can be determined; the preprocessing operation is performed on each type of image, the channel number of each type of image is set to the maximum channel number, and the resolution of each type of image is adjusted to a preset resolution corresponding to each type. A preprocessing parameter modification interface can also be provided for the preprocessing operation to enable a user to adjust the parameter information of the preprocessing operation through the preprocessing parameter modification interface.

[0104] In another example embodiment of the present disclosure, the image acquisition module 710 can acquire a plurality of product information corresponding to the detection task; acquire images corresponding to each product information, wherein the resolution and the channel number of each type of image are the same.

[0105] The model acquisition module 720 can acquire a plurality of defect detection models corresponding to each type of image, wherein the plurality of defect detection models are trained from the same initial model.

[0106] The defect detection module 730 is configured to perform defect detection on each type of image by using the defect detection model corresponding to each type of image to obtain a defect detection result.

[0107] Wherein, the defect detection model corresponding to each type of image is trained from the same initial model.

[0108] In an example embodiment of the present disclosure, the defect detection model includes a feature extraction network and a defect recognition network. The defect detection module 730 can use the feature extraction network to extract features from various types of images to obtain feature images; and standardize the feature images; use the defect recognition network to classify defects based on the standardized feature images to obtain defect categories of each feature image, and determine the coordinates of each feature image; and obtain defect detection results based on the defect category and coordinates.

[0109] In this exemplary embodiment, the defect detection module 730 may further be configured with a standardization processing parameter adjustment interface, so that the user can adjust the parameter information of the standardization processing through the standardization processing parameter adjustment interface.

[0110] In this example embodiment, when obtaining defect detection results based on defect categories and coordinates, the defect detection module 730 may filter the feature images to obtain a target feature image, determine the defect category and coordinates of the target candidate feature image, and obtain the defect detection results based on a preset filtering strategy and the defect category and coordinates of the target feature image. Furthermore, the defect detection module 730 may calculate the confidence level of each feature image and, based on the confidence level, filter the feature images using the NMS algorithm to obtain the target feature image.

[0111] In this example embodiment, the defect detection module 730 can also set defect impact weights for various types of images based on configuration parameters; use the defect detection model corresponding to the type of each image to perform defect detection on various types of images to obtain reference defect detection results corresponding to each type of image; and determine the defect detection results based on the defect impact weights and the reference defect detection results.

[0112] The specific details of each module in the above device have been described in detail in the implementation method part. The undisclosed details can be found in the implementation method part, so they will not be repeated here.

[0113] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0114] The exemplary embodiments of the present disclosure further provide a computer-readable storage medium having stored thereon a program product capable of implementing the methods described above in this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code that, when executed on a terminal device, causes the terminal device to execute the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of the present disclosure.

[0115] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0116] In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0117] In addition, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0118] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0119] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A defect detection method, characterized in that: include: Acquire a detection task and acquire multiple types of images corresponding to the detection task; Obtaining defect detection models corresponding to the types of the images, respectively, obtained by training with the same initial model; Performing defect detection on various types of images using the defect detection model corresponding to each type of image to obtain defect detection results; The multiple types of images corresponding to the detection task are obtained, including: Obtain product information corresponding to the detection task; Acquire multiple types of images of the same product corresponding to the product information; wherein the various types of images are obtained by photographing the product using cameras with different configuration parameters; determining the maximum number of channels in each of the described types of images; For each type of image, performing a preprocessing operation on the image of the type to set the number of channels of each type of image to the maximum number of channels; Furthermore, for each type of image, a resolution threshold is determined according to the defect detection model, and the resolution of the type of image is adjusted according to the resolution threshold, so as to scale each type of image to a different preset resolution.

2. The method according to claim 1, characterized in that The configuration parameters include one or more of resolution, color, and magnification.

3. The method according to claim 2, characterized in that The various types of images include one or more of an AOI color map, a TDI map, and a DM map of the product.

4. The method according to claim 2, characterized in that The defect detection is performed on various types of images using the defect detection model corresponding to each type of image to obtain a defect detection result, including: Setting defect impact weights for the various types of images according to the configuration parameters; Perform defect detection on various types of images using the defect detection model corresponding to each type of image to obtain reference defect detection results corresponding to each type of image; The defect detection result is determined according to the defect impact weight and the reference defect detection result.

5. The method according to claim 1, wherein Obtain product information corresponding to the inspection task, including: Extract the product information field in the detection task to obtain the product information.

6. The method according to claim 1, characterized in that The method further comprises: A preprocessing parameter modification interface is provided for the preprocessing operation so that the user can adjust the parameter information of the preprocessing operation through the preprocessing parameter modification interface.

7. The method according to claim 1, characterized in that The defect detection result includes normal image information or image defect information.

8. The method according to claim 1, characterized in that The defect detection model includes a feature extraction network and a defect recognition network. The defect detection model corresponding to each type of image is used to perform defect detection on various types of images to obtain defect detection results, including: Using the feature extraction network to extract features from various types of images to obtain feature images; and performing standardization processing on the feature images; Using the defect recognition network to classify defects according to the standardized feature images, obtain defect categories for each feature image, and determine the coordinates of each feature image; The defect detection result is obtained according to the defect category and the coordinates.

9. The method according to claim 8, characterized in that The method further comprises: A standardization processing parameter adjustment interface is configured to enable a user to adjust the parameter information of the standardization processing through the standardization processing parameter adjustment interface.

10. The method according to claim 8, characterized in that Obtaining the defect detection result according to the defect category and the coordinates includes: Screening the characteristic image to obtain a target characteristic image, and determining the defect category and the coordinates of the target candidate characteristic image; The defect detection result is obtained according to a preset screening strategy and the defect category and the coordinates of the target feature image.

11. The method according to claim 10, characterized in that The screening of the characteristic image to obtain a target characteristic image includes: Calculating the confidence of each feature image; The feature image is screened using the NMS algorithm according to the confidence level to obtain a target feature image.

12. A defect detection device, characterized in that: include: An image acquisition module is used to acquire a detection task and obtain multiple types of images corresponding to the detection task; A model acquisition module, configured to acquire defect detection models corresponding to the types of the images and obtained by training with the same initial model; A defect detection module, configured to perform defect detection on various types of images using the defect detection model corresponding to each type of image to obtain defect detection results; Wherein, the image acquisition module is further configured to execute: Obtain product information corresponding to the detection task; Acquire multiple types of images of the same product corresponding to the product information; wherein the various types of images are obtained by photographing the product using cameras with different configuration parameters; determining the maximum number of channels in each of the described types of images; For each type of image, performing a preprocessing operation on the image of the type to set the number of channels of each type of image to the maximum number of channels; Furthermore, for each type of image, a resolution threshold is determined according to the defect detection model, and the resolution of the type of image is adjusted according to the resolution threshold, so as to scale each type of image to a different preset resolution.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the defect detection method according to any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: include: processor; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the defect detection method according to any one of claims 1 to 11.

Images