Image defect recognition

Abstract

A method, device, and computer program product for image processing are provided. In this method, it is determined whether a first image indicates a defect associated with a target object. In response to determining that the first image indicates a defect, a second image without the defect is obtained based on the first image. The defect is identified by comparing the first image and the second image. In this way, the defect associated with the target object in the image can be accurately and effectively identified or segmented.

Description

Background Art

[0001] The present invention relates to image processing, and more particularly, to a method, an apparatus, and a computer program product for identifying defects associated with a target object in an image.

[0002] Nowadays, there is a high demand for automatic and accurate defect segmentation in the manufacturing industry. In the manufacturing industry, for example, in fields such as manual inspection of smartphone part assembly, component-level defect inspection on printed circuit boards (PCBs) (more than 20 types of defects), and liquid crystal display (LCD) panel defect detection (more than 120 types of defects), the demand for automatic vision detection technology for defect segmentation is increasing.

[0003] Accurate defect segmentation is of great value for determining defect severity and subsequent processing procedures (such as repair, rework, ignore, disposal, etc.). Due to the large amount of work required, inspectors are more willing to perform defect annotation at the image level (for example, annotating each image with defect type labels) rather than precisely determining defect locations or performing annotations at the pixel level. Summary of the Invention

[0004] According to an embodiment of the present invention, there is provided a method for image processing. In this method, it is determined whether a first image indicates a defect associated with a target object. In response to determining that the first image indicates a defect, a second image without defects is obtained based on the first image. The defect is identified by comparing the first image and the second image.

[0005] According to another embodiment of the present invention, there is provided an apparatus for image processing. The apparatus includes a processing unit and a memory coupled to the processing unit and storing instructions thereon. When executed by the processing unit, the instructions perform the following actions: determining whether a first image indicates a defect associated with a target object; in response to determining that the first image indicates a defect, obtaining a second image without defects based on the first image; and identifying the defect by comparing the first image and the second image.

[0006] According to still another embodiment of the present invention, there is provided a computer program product, which is tangibly stored on a non-transitory machine-readable medium and includes machine-executable instructions. When executed on a device, the instructions cause the device to perform actions including: determining whether a first image indicates a defect associated with a target object; in response to determining that the first image indicates a defect, obtaining a second image without defects based on the first image; and identifying the defect by comparing the first image and the second image. Brief Description of the Drawings

[0007] Preferred embodiments of the present invention will now be described by way of example only and with reference to the following drawings:

[0008] Figure 1 Represents a cloud computing node according to an embodiment of the present invention.

[0009] Figure 2 Represents a cloud computing environment according to an embodiment of the present invention.

[0010] Figure 3 Represents an abstract model layer according to an embodiment of the present invention.

[0011] Figure 4 Shows a flowchart of an example method for image processing according to an embodiment of the present invention.

[0012] Figure 5 Shows a schematic diagram of an example defect segmentation according to an embodiment of the present invention.

[0013] Figure 6 Shows a schematic diagram of another example defect segmentation according to an embodiment of the present invention.

[0014] Figure 7 Shows a schematic diagram of yet another example defect segmentation according to an embodiment of the present invention.

[0015] In the embodiments of the present disclosure, the same reference numerals generally refer to the same components. Detailed Description of the Invention

[0016] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. Although the embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited by the embodiments set forth herein.

[0017] First of all, it should be understood that although the present disclosure includes a detailed description of cloud computing, the implementation of the technical solutions described therein is not limited to a cloud computing environment, but can be implemented in conjunction with any other type of computing environment known or developed in the future.

[0018] Cloud computing is a service delivery model for convenient, on-demand network access to a shared pool of configurable computing resources. Configurable computing resources are resources that can be quickly deployed and released with minimal management cost or minimal interaction with the service provider, such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0019] The characteristics include:

[0020] On-demand self-service: Cloud consumers can unilaterally and automatically deploy computing capabilities such as server time and network storage on demand without human interaction with the service provider.

[0021] Broad network access: Computing capabilities can be obtained over a network via standard mechanisms that promote the use of the cloud through various types of thin client platforms or thick client platforms (such as mobile phones, laptops, personal digital assistants PDA).

[0022] Resource pooling: The provider's computing resources are pooled and served to multiple consumers through a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated on demand. In general, consumers do not control or even know the exact location of the provided resources, but can specify the location at a higher level of abstraction (such as country, state, or data center), thus having location independence.

[0023] Rapid elasticity: It is possible to rapidly and elastically (sometimes automatically) deploy computing capabilities to enable rapid scaling out and rapidly release to scale in quickly. To the consumer, the available computing capabilities for deployment often appear to be infinite and any amount of computing capabilities can be obtained at any time.

[0024] Measured service: The cloud system automatically controls and optimizes resource utility by leveraging metering capabilities at some level of abstraction appropriate to the service type (such as storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the service provider and consumers.

[0025] The service models are as follows:

[0026] Software as a Service (SaaS): The ability provided to consumers is to use applications that the provider runs on the cloud infrastructure. The applications can be accessed from various client devices through a thin client interface such as a web browser (such as web-based email). Except for limited application configuration settings specific to the user, consumers neither manage nor control the underlying cloud infrastructure, including the network, servers, operating systems, storage, and even individual application capabilities.

[0027] Platform as a Service (PaaS): The ability provided to consumers is to deploy applications created or acquired by the consumers on the cloud infrastructure, and these applications are created using programming languages and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but have control over the deployed applications and may also have control over the application hosting environment configuration.

[0028] Infrastructure as a Service (IaaS): The ability provided to the consumer is the ability to deploy and run arbitrary software, including operating systems and applications, on processing, storage, networks, and other fundamental computing resources. The consumer neither manages nor controls the underlying cloud infrastructure, but has control over the operating systems, storage, and the applications deployed thereon, and may have limited control over the selected network components (such as a host firewall).

[0029] The deployment models are as follows:

[0030] Private cloud: The cloud infrastructure runs solely for a particular organization. The cloud infrastructure can be managed by the organization or a third party and can exist either within or outside the organization.

[0031] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with common interests (such as mission, security requirements, policies, and compliance considerations). The community cloud can be managed by multiple organizations within the community or a third party and can exist either within or outside the community.

[0032] Public cloud: The cloud infrastructure is provided to the public or a large industrial group and is owned by the organization selling the cloud services.

[0033] Hybrid cloud: The cloud infrastructure consists of two or more clouds of different deployment models (private cloud, community cloud, or public cloud), which remain distinct entities but are bound together by standardized or proprietary technologies that enable the portability of data and applications (such as cloud bursting traffic sharing technology for load balancing between clouds).

[0034] The cloud computing environment is service-oriented, with characteristics focused on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0035] Now refer to Figure 1 , which shows an example of a cloud computing node. Figure 1 The shown cloud computing node 10 is merely an example of a suitable cloud computing node and should not impose any limitations on the functions and scope of use of the embodiments of the present invention. In general, the cloud computing node 10 can be used to implement and / or execute any of the functions described above.

[0036] The cloud computing node 10 has a computer system / server 12 or a mobile electronic device (such as a communication device), which can operate with numerous other general-purpose or special-purpose computing system environments or configurations. As is well known, examples of computing systems, environments, and / or configurations suitable for operating with the computer system / server 12 include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and so on.

[0037] The computer system / server 12 can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer system / server 12 can be implemented in a distributed cloud computing environment where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0038] As Figure 1 shown, the computer system / server 12 in the cloud computing node 10 is presented in the form of a general-purpose computing device. The components of the computer system / server 12 can include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0039] The bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0040] The computer system / server 12 typically includes a variety of computer system-readable media. These media can be any accessible media by the computer system / server 12, including volatile and non-volatile media, removable and non-removable media.

[0041] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 1 not shown, commonly referred to as a "hard disk drive"). Although Figure 1 not shown in the figure, a disk drive for reading and writing on removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing on removable non-volatile optical disks (such as CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0042] A program / utility 40 having a set (at least one) of program modules 42 can be stored in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples. Program modules 42 generally perform the functions and / or methods in the embodiments described in the present invention.

[0043] The computer system / server 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the computer system / server 12, and / or communicate with any device that enables the computer system / server 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the computer system / server 12 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the computer system / server 12 through the bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can operate with the computer system / server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0044] Now refer to Figure 2, which shows an exemplary cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud computing consumers can communicate. The local computing devices can be, for example, a personal digital assistant (PDA) or a mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an in-vehicle computer system 54N. The cloud computing nodes 10 can communicate with each other. The cloud computing nodes 10 can be physically or virtually grouped in one or more networks including, but not limited to, a private cloud, a community cloud, a public cloud, or a hybrid cloud as described above, or a combination thereof (not shown in the figure). In this way, the consumers of the cloud can request infrastructure as a service (IaaS), platform as a service (PaaS), and / or software as a service (SaaS) provided by the cloud computing environment 50 without maintaining resources on the local computing devices. It should be understood that Figure 2 The various computing devices 54A-N shown are merely illustrative, and the cloud computing nodes 10 and the cloud computing environment 50 can communicate with any type of computing device (e.g., using a web browser) on and / or network-addressably connected to any type of network.

[0045] Now refer to Figure 3 , which shows a set of functional abstraction layers provided by the cloud computing environment 50 ( Figure 2 ). First, it should be understood that Figure 3 The components, layers, and functions shown are merely illustrative, and embodiments of the present invention are not limited thereto. As Figure 3 shown, the following layers and corresponding functions are provided:

[0046] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61; a server 62 based on the RISC (reduced instruction set computer) architecture; a server 63; a blade server 64; a storage device 65; a network and network components 66. Examples of software components include: network application server software 67 and database software 68.

[0047] The virtualization layer 70 provides an abstraction layer that can provide examples of the following virtual entities: virtual servers 71, virtual storage 72, virtual networks 73 (including virtual private networks), virtual applications and operating systems 74, and virtual clients 75.

[0048] In one example, the management layer 80 may provide the following functions: Resource provisioning function 81: Provide for the dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment; Metering and pricing function 82: Track the cost of resource usage within the cloud computing environment and provide bills and invoices therefor. In one example, the resources may include application software licenses. Security function: Provide authentication for cloud consumers and tasks and provide protection for data and other resources. User portal function 83: Provide access to the cloud computing environment for consumers and system administrators. Service level management function 84: Provide the allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment function 85: Provide advanced arrangement and provision for future requirements of cloud computing resources predicted according to the SLA.

[0049] The workload layer 90 provides examples of functions that a cloud computing environment may implement. Examples of workloads or functions that may be provided in this layer include: Mapping and navigation 91; Software development and lifecycle management 92; Provision of teaching in a virtual classroom 93; Data analysis processing 94; Transaction processing 95; and Image processing 96.

[0050] As described above, there is a need to be able to perform defect segmentation based on weakly supervised image-level annotations. Traditionally, fully convolutional networks (FCNs) have been used for semantic segmentation. However, FCNs require precise object localization at the pixel granularity and high labeling effort in the preparation of training data.

[0051] Additionally, attention-based methods can be used for pixel-wise classification. In attention-based methods, object segmentation is trained using image labels. Visualization methods (such as class activation heatmaps) from pre-trained convolutional neural network (CNN) classification models can be used to indicate which regions in the image are relevant to the class. However, the heatmaps only roughly cover the objects of the class. In addition, when there are many objects of the same class in the image, only one or a few objects are indicated in the heatmap.

[0052] To at least partially address one or more of the above problems and other potential problems, example embodiments of the present disclosure propose a solution for image processing.

[0053] Generally, according to embodiments of the present disclosure, an image of a target object (referred to as the "first image") can be obtained. The first image may present a specific pattern or periodic feature, but is not limited thereto. For example, the first image can be an image of an LCD / PCB panel. Then the first image can be applied to an image classifier to classify the image as normal or abnormal. A normal image can represent an image of a target object that does not include defects, and an abnormal image can represent an image of a target object that includes at least one defect. For example, the defect can be a bad pixel, scratch, bubble, etc. in an LCD panel, or poor soldering, missing components, etc. of a PCB panel.

[0054] When the classification result indicates that the first image indicates a defect, a heat map for localizing the defect in the first image can be generated, and a mask covering at least a part of the defect can be generated based on the heat map. Then, by removing at least a part of the defect covered by the mask from the first image, a restored image can be generated. For example, the restored image can be generated by applying the image to be restored with the mask to a Generative Adversarial Network (GAN)-based model trained to restore defects.

[0055] The restored image can be applied to the image classifier. If the classification result indicates that the restored image does not include defects, that is, the restored image is a completely restored normal image (referred to as the "second image"), the defect in the first image can be identified by comparing the first image and the second image. Otherwise, the restored image will iteratively undergo further restoration processes until a completely restored normal image is obtained, so that the defect in the first image can be identified. It should be understood that in the context of the present invention, identifying a defect in an image means performing defect segmentation on the image.

[0056] In this way, defects associated with the target object can be automatically and accurately segmented from the image. Since the image classification and restoration processes for defect segmentation are performed at the image level, defect segmentation can be used for weakly supervised or image-level annotation datasets. In this case, pixel-level segmentation labels are not required for training, thus significantly saving location marking in traditional object detection / image segmentation tasks. Additionally, the defect segmentation of the present invention can be widely applied. It is not limited to being applied to images with rigid patterns or templates, and can be applied to images with various defect quantities or sizes. Furthermore, the result of defect segmentation can also be used as segmentation annotation. Therefore, the proposed solution improves the accuracy, efficiency, and applicability of defect segmentation and improves the user experience in defect inspection.

[0057] Now reference will be made to Figures 4 - 7 describe some example embodiments. Figure 4A flowchart of an example method 400 for image processing according to an embodiment of the present invention is shown. Method 400 can be implemented at least in part by computer system / server 12 or other suitable systems. Figure 5 A schematic diagram of an example defect segmentation 500 according to an embodiment of the present invention is shown. For the purpose of discussion, reference will be made to Figure 5 Describe method 400.

[0058] At 410, computer system / server 12 determines whether the first image indicates a defect associated with the target object. In some embodiments, the first image may present a specific pattern or periodic feature. For example, the first image can be an image of an LCD / PCB panel. An example of the first image is shown as Figure 5 Image 510 in.

[0059] In some embodiments, computer system / server 12 can obtain the first image and apply the first image to an image classifier for classifying the image as normal or abnormal. A normal image can represent an image of a target object that does not include a defect, and an abnormal image can represent an image of a target object that includes a defect. The image classifier can be any suitable image classifier, such as but not limited to a binary classifier model. Since the binary classifier model is trained to identify whether an image is normal or abnormal, a weakly supervised or image-level annotation dataset is sufficient.

[0060] As an example, during the training phase of the image classifier, a large number of images can be applied to train the image classifier. Some images can be labeled as normal images, while other images can be labeled as abnormal images indicating defects. No location information about the defects needs to be provided. After the training process, the image classifier can achieve high classification accuracy.

[0061] If the classification result indicates that the first image is abnormal, computer system / server 12 can determine that the first image indicates a defect. Otherwise, it is determined that the first image does not represent a defect.

[0062] At 420, if the first image indicates a defect, computer system / server 12 obtains a second image that does not have a defect based on the first image. An example of the second image is shown as Figure 5 Image 540 in.

[0063] In some embodiments, to generate the second image, the computer system / server 12 may generate a heat map indicating the heat values of the pixels in the first image. Specifically, the heat map can be generated by applying the first image to a class activation heat map model that is trained to locate defects in the first image. Generally, the higher the heat value of a pixel, the more likely the pixel is related to a defect. In this case, the heat map can locate the defects in the first image. An example of the heat map is shown in the image 520 in Figure 5 shown.

[0064] However, as Figure 5 shown, the heat map only roughly locates the defects in the first image. In this case, the computer system / server 12 may generate a mask covering at least a portion of the defects based on the heat map. An example of the mask is shown in the image 530 in Figure 5 shown.

[0065] The mask can generally have a predetermined shape, such as a square or a rectangle, so that the mask can more precisely locate the defects for later defect removal. This is because the size of the mask directly affects the defect removal or image restoration performance. Generally, a smaller mask can produce better performance.

[0066] In some embodiments, the computer system / server 12 may determine that the heat values of a group of pixels in the first image exceed a predetermined threshold and generate a mask covering the group of pixels. In this way, the mask can cover the most suspicious defect areas.

[0067] Then, the computer system / server 12 may produce the second image by removing at least a portion of the defects covered by the mask from the first image. The defects can be removed by filling the pixels in the masked area of the image with reasonable pixels. For example, the second image can be generated by applying the first image with the mask to a model based on a generative adversarial network (GAN) that is trained to remove at least part of the defects.

[0068] The purpose of the GAN-based model is to generate a masked area that looks real and natural and is similar to the unmasked original image. To achieve this purpose, a large number of normal images can be used to train the GAN-based model. The training can be performed by randomly masking areas in the normal images. The loss function of the GAN-based model is the sum of the pixel reconstruction loss and the adversarial discriminator loss. Visually, the GAN-based model is applicable to most cases, which results in satisfactory segmentation results. Additionally, since the GAN-based model is trained from normal images, the GAN-based model can be considered weakly supervised or even unsupervised.

[0069] In addition to GAN-based models, other image restoration techniques for defect removal can also be used, such as matching and copying background patches to masked regions, or matching masked regions from a database with image indices. For example, by indexing corresponding normal images in the database and copying the corresponding regions in the indexed images to the masked regions.

[0070] Furthermore, due to the need to handle various defect quantities or sizes, in some embodiments, defects cannot be removed in one go. For example, the first image can be an image of a target object with more than one defect, or an image of a target object with a large defect. In these embodiments, the first image needs to be iteratively restored to obtain a second image without defects.

[0071] For example, to determine whether the restored image generated from the first image is defect-free, the computer system / server 12 can obtain a restored image (referred to as an "intermediate image") by removing at least a portion of the defects from the first image. The computer system / server 12 can determine whether the intermediate image indicates that the defects have been completely removed from the intermediate image. If so, the computer system / server 12 can directly determine the intermediate image as the second image.

[0072] Otherwise, if the defects are not completely removed from the intermediate image, for example, a remaining portion of the defects still exists in the intermediate image, the computer system / server 12 can remove the remaining portion of the defects from the intermediate image based on the heatmap of the intermediate image. Specifically, the computer system / server 12 can generate a heatmap indicating the heat values of the pixels in the intermediate image, and generate a mask covering at least a portion of the remaining defects based on the heatmap. Then, the computer system / server 12 can generate another image by removing at least a portion of the remaining defects covered by the mask. The computer system / server 12 can again determine whether the additional image indicates that the defects are completely removed. In this way, the image restoration process is iteratively repeated until a second image without defects is obtained. Such iterative restoration can ensure segmentation integrity.

[0073] Reference Figure 6 and Figure 7 describe more detailed examples regarding images of target objects with large defects and images of target objects with more than one defect.

[0074] After obtaining the second image without defects, at 430, the computer system / server 12 can identify the defects by comparing the first image and the second image. In this way, the defects can be segmented from the first image. For example, the computer system / server 12 can use simple subtraction or mathematical morphology methods to obtain the segmentation result from the first image and the second image. Additionally, the segmentation result can also be used as segmentation annotation for further processing. As an example of the segmentation result,Figure 5 The image 550 shows the segmented defect.

[0075] In this way, by combining the binary classifier model, the class activation heatmap model, and the GAN-based model, defect segmentation can be performed on weakly supervised or image-level annotation datasets. Additionally, this defect segmentation can be widely applied and can be used for various numbers or sizes of defects. Therefore, the proposed solution improves the accuracy, efficiency, and applicability of defect segmentation and enhances the user experience in defect inspection.

[0076] Figure 6 FIG. shows a schematic diagram of an exemplary defect segmentation 600 of an image including a large defect according to an embodiment of the present invention. The defect segmentation 600 can be at least partially implemented by the computer system / server 12 or other suitable systems.

[0077] As Figure 6 shown, the first image 610 includes an extended defect associated with the target device. The computer system / server 12 determines that the first image 610 indicates at least one defect. Next, the computer system / server 12 generates a heatmap 612 that locates the defect in the first image 610. It can be seen that only a part of the defect is emphasized in the heatmap 612. In this case, the computer system / server 12 generates a mask 614 that only covers the upper part of the defect. Then, the computer system / server 12 generates an intermediate image 720 by removing the upper part of the defect covered by the mask 614.

[0078] However, the remaining part of the defect is not removed from the intermediate image 620, and the intermediate image 620 is not fully restored. In this case, the computer system / server 12 determines that the intermediate image 620 indicates at least one defect and performs another iteration to remove the defect. That is, the computer system / server 12 generates a heatmap 622 that locates the remaining defect and a mask 624 that covers the upper part of the remaining defect. Then, the computer system / server 12 generates another image by removing the upper part of the remaining defect covered by the mask 624. The iteration for removing the defect is repeated until a second image 630 that excludes any defect is obtained.

[0079] The computer system / server 12 identifies or segments the defect by comparing the first image 610 and the second image 630. The defect segmentation result is shown in the image 640, which shows the segmented extended defect. In this way, by restoring the defect image and comparing the original image with the restored image, defects with large sizes or lengths in the image can be accurately segmented.

[0080] Figure 7A schematic diagram of an example defect segmentation 700 of an image including multiple defects according to an embodiment of the present invention is shown. The defect segmentation 700 can be implemented at least in part by a computer system / server 12 or other suitable systems.

[0081] As Figure 7 shown, the first image 710 includes two defects associated with the target device. The computer system / server 12 determines that the first image 710 indicates at least one. Next, the computer system / server 12 generates a heat map 712 that locates the defects in the first image 710. The heat map 712 locates the two defects, but the lower defect is emphasized in the heat map 712. In this case, the computer system / server 12 generates a mask 714 that only covers the lower defect. Then, the computer system / server 12 generates an intermediate image 720 by removing the lower defect covered by the mask 714.

[0082] However, the upper defect is not removed from the intermediate image 720, and the intermediate image 720 is not fully restored. In this case, the computer system / server 12 determines that the intermediate image 720 indicates at least one defect and performs another iteration to remove the defect. That is, the computer system / server 12 generates a heat map 722 that locates the upper defect and a mask 724 that covers the upper defect. Then, the computer system / server 12 produces a second image 730 by removing the upper defect covered by the mask 724.

[0083] Finally, the computer system / server 12 determines that the second image 730 does not include any defects and identifies or segments the defects by comparing the first image 710 with the second image 730. The defect segmentation result is shown in FIG. 740, which shows two segmented defects. In this way, multiple defects in the image can be accurately segmented.

[0084] At any possible level of combination of technical details, the present invention can be a system, a method, and / or a computer program product. The computer program product can include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0085] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or can be downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0087] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present invention.

[0088] Aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions.

[0089] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner, so that the computer - readable medium storing the instructions includes a manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0092] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the technical improvement of technologies in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for image processing, comprising: Determining, by one or more processors, whether a first image indicates a defect associated with a target object; In response to determining that the first image indicates the defect, generating, by one or more processors, a mask covering at least a portion of the defect based on a heat map; Generating, by one or more processors, a second image by removing at least a portion of the defect covered by the mask from the first image; And Identifying, by one or more processors, the defect by comparing the first image and the second image.

2. The computer-implemented method according to claim 1, wherein determining whether the first image indicates the defect comprises: Applying, by one or more processors, the first image to an image classifier configured to classify an image as normal or abnormal; And In response to the classification result indicating that the first image is abnormal, determining, by one or more processors, that the first image indicates the defect.

3. The computer-implemented method according to claim 2, wherein, The image classifier is a binary classification model.

4. The computer-implemented method according to claim 1, wherein, The heat map indicates the heat values of pixels in the first image.

5. The computer-implemented method according to claim 1, further comprising: Generating, by one or more processors, the heat map by applying the first image to a class activation heat map model trained to locate the defect in the first image.

6. The computer-implemented method according to claim 1, wherein, Generating the mask comprises: Determining, by one or more processors, that heat values of a set of pixels in the first image exceed a predetermined threshold; and Generating, by one or more processors, the mask covering the set of pixels.

7. The computer-implemented method according to claim 1, wherein generating the second image comprises: Generating, by one or more processors, the second image by applying the first image with the mask to a generative adversarial network (GAN)-based model trained to remove at least a portion of the defect.

8. The computer-implemented method according to claim 1, wherein generating the second image comprises: Obtaining, by one or more processors, an intermediate image by removing at least a portion of the defect from the first image; In response to the defect not being present in the intermediate image, determining, by one or more processors, the intermediate image as the second image; And In response to a remaining portion of the defect being present in the intermediate image, removing, by one or more processors, the remaining portion from the intermediate image based on a heat map of the intermediate image to generate the second image.

9. An image processing apparatus, characterized in that, Comprising: A processing unit; And A memory coupled to the processing unit and storing instructions thereon, the instructions, when executed by the processing unit, perform actions, the actions including: Determining whether a first image indicates a defect associated with a target object; In response to determining that the first image indicates the defect, generating a mask covering at least a portion of the defect based on a heat map; Generating a second image by removing at least a portion of the defect covered by the mask from the first image; and Identifying the defect by comparing the first image with the second image.

10. The apparatus according to claim 9, wherein determining whether the first image indicates the defect comprises: applying the first image to an image classifier that is configured to classify an image as normal or abnormal; and determining that the first image indicates the defect in response to the classification result indicating that the first image is abnormal.

11. The apparatus according to claim 10, wherein, The image classifier is a binary classification model.

12. The device according to claim 9, wherein, The heat map indicates the heat value of pixels in the first image.

13. The device according to claim 9, wherein, The operation further comprises: generating the heat map by applying the first image to a class activation heat map model trained to locate the defect in the first image.

14. The apparatus according to claim 9, wherein Generating the mask comprises: determining that the heat value of a set of pixels in the first image exceeds a predetermined threshold; and generating the mask covering the set of pixels.

15. The device according to claim 9, wherein, Generating the second image comprises: generating the second image by applying the first image with the mask to a generative adversarial network (GAN)-based model trained to remove at least a portion of the defect.

16. The apparatus according to claim 9, wherein, Generating the second image comprises: obtaining an intermediate image by removing at least a portion of the defect from the first image; determining the intermediate image as the second image in response to the absence of the defect in the intermediate image; and generating the second image by removing the remaining portion from the intermediate image based on the heat map of the intermediate image in response to the presence of the remaining portion of the defect in the intermediate image.

17. A computer program product comprising program instructions executable by a processor to cause the processor to perform the following operations: determining whether a first image indicates a defect associated with a target object; generating, in response to determining that the first image indicates the defect, a mask covering at least a portion of the defect based on a heat map; generating a second image by removing at least the portion of the defect covered by the mask from the first image; and identifying the defect by comparing the first image with the second image.

18. The computer program product according to claim 17, wherein determining whether the first image indicates the defect comprises: applying the first image to an image classifier that is configured to classify an image as normal or abnormal; and determining that the first image indicates the defect in response to the classification result indicating that the first image is abnormal.

19. The computer program product according to claim 17, wherein, The heat map indicates the heat value of pixels in the first image.

20. The computer program product according to claim 17, wherein, Generating the second image comprises: obtaining an intermediate image by removing at least a portion of the defect from the first image; determining the intermediate image as the second image in response to the absence of the defect in the intermediate image; and generating the second image by removing the remaining portion from the intermediate image based on the heat map of the intermediate image in response to the presence of the remaining portion of the defect in the intermediate image.

21. A computer-readable storage medium storing program code that, when run on a computer, is adapted to execute the method of any one of claims 1 to 8.

Images