Method, model training method and apparatus for defect detection

Abstract

The application provides a defect detection method and a model training method and device, and relates to the technical field of detection. The defect detection method comprises the following steps: acquiring a to-be-detected image, wherein the to-be-detected image comprises a product image; inputting the to-be-detected image into a region segmentation model to obtain a region segmentation result, wherein the region segmentation result comprises a to-be-measured region; wherein the to-be-measured region comprises a region in the product image that needs to be detected for defects, and does not comprise a region in the product image that does not need to be detected for defects; acquiring a to-be-measured region image according to the to-be-measured region; and inputting the to-be-measured region image into a defect detection model to obtain a first defect detection result. The embodiments of the application can be beneficial to distinguishing defects in the to-be-measured region of the product, thereby improving the reliability of defect detection.

Description

Technical Field

[0001] This application relates to the field of industrial inspection technology, and in particular to a method for defect detection, a method for model training, and an apparatus. Background Technology

[0002] With the rapid development of Artificial Intelligence (AI) technology and increasingly fierce competition in the industrial market, AI-based industrial quality inspection is being applied to industrial processes to reduce labor costs without affecting product quality and economic efficiency. AI-based industrial quality inspection provides support for process control, quality control, process management, and decision-making, further realizing quality control and process optimization of manufactured products. Machine learning-based defect detection models can detect appearance defects in products using machine vision, thus achieving AI-based industrial quality inspection.

[0003] In related defect detection technologies, a full-area product image is captured and input into a machine learning-based defect detection model for full-area defect detection. During training, the model labels defects in the target area of ​​the product, omitting defects in other parts of the product. However, because the model can only distinguish whether something is a defect, but not the specific area where the defect is located, it may output false positives outside the target area during testing, resulting in low reliability of the defect detection method. Therefore, improving the reliability of defect detection is a pressing issue. Summary of the Invention

[0004] This application provides a method for defect detection, a method for model training, and an apparatus that can help improve the reliability of defect detection.

[0005] In a first aspect, embodiments of this application provide a defect detection method, including:

[0006] Acquire an image to be detected, the image to be detected including a product image;

[0007] The image to be detected is input into a region segmentation model to obtain a region segmentation result, which includes the region to be tested; wherein, the region to be tested includes the region in the product image that needs to be detected for defects, but does not include the region in the product image that does not need to be detected for defects.

[0008] Based on the area to be tested, obtain an image of the area to be tested;

[0009] The image of the region to be tested is input into the defect detection model to obtain the first defect detection result;

[0010] The region segmentation model and the defect detection model are respectively trained on the neural network model based on the product sample image.

[0011] Secondly, embodiments of this application provide a method for model training, including:

[0012] Obtain a training sample set, which includes sample images of the product and annotation information of the test area of ​​the sample images. The test area includes the area of ​​the product image that needs to be detected for defects, but does not include the area of ​​the product image that does not need to be detected for defects.

[0013] The sample image is input into the region segmentation model to obtain the predicted region segmentation result, which includes the predicted region to be tested.

[0014] The model loss is determined based on the predicted region segmentation results and the annotation information of the test region in the sample image;

[0015] Based on the model loss, the parameters of the region segmentation model are updated to obtain the trained model.

[0016] Thirdly, embodiments of this application provide a defect detection apparatus, comprising:

[0017] The first acquisition unit is used to acquire an image to be detected, the image to be detected including a product image;

[0018] A region segmentation model is used as input to the image to be detected to obtain a region segmentation result, wherein the region segmentation result includes the region to be detected; wherein the region to be detected includes the region in the product image that needs to be detected for defects, but does not include the region in the product image that does not need to be detected for defects;

[0019] The second acquisition unit is used to acquire an image of the region to be tested based on the region to be tested.

[0020] A defect detection model is used to input the image of the region to be tested into the defect detection model to obtain a first defect detection result;

[0021] The region segmentation model and the defect detection model are respectively trained on the neural network model based on the product sample image.

[0022] Fourthly, embodiments of this application provide a model training apparatus, comprising:

[0023] The acquisition unit is used to acquire a training sample set, which includes sample images of the product and annotation information of the test area of ​​the sample images. The test area includes the area of ​​the product image that needs to be detected for defects, but does not include the area of ​​the product image that does not need to be detected for defects.

[0024] A region segmentation model is used to take the sample image as input and obtain a predicted region segmentation result, wherein the predicted region segmentation result includes a predicted region to be tested.

[0025] The determining unit is used to determine the model loss based on the predicted region segmentation result and the annotation information of the test region of the sample image;

[0026] The update unit is used to update the parameters of the region segmentation model according to the model loss to obtain the trained model.

[0027] Fifthly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the methods as described in the first or second aspect.

[0028] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when read and executed by a processor of a computer device, cause the computer device to perform the methods described in the first or second aspect.

[0029] In a seventh aspect, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the first or second aspect.

[0030] Through the above technical solution, this embodiment of the application obtains the test region of the image to be inspected by performing region segmentation. Since the test region includes areas in the product image that need to be detected for defects, and excludes areas in the product image that do not need to be detected for defects, the test region image obtained based on the test region is input into the defect detection model. The resulting defect detection result includes the defect bounding box and defect type of the test region, which helps to distinguish defects in the product's test region and reduce false detections outside the test region output by the model. Because this embodiment of the application considers the location of the defect, it conforms to the judgment rules for defect detection in industrial applications, which helps to improve the reliability of defect detection.

[0031] Furthermore, in this embodiment, the image of the region to be tested is obtained based on the region to be tested, and the image of the region to be tested is input into the defect detection model, which can help increase the proportion of defects in the whole image and amplify the defect features, thereby helping to improve the accuracy of defect identification.

[0032] Furthermore, in the defect detection embodiment of this application, only the image of the region to be detected needs to be processed, instead of the entire image, which helps to reduce the waste of hardware resources and reduce the training and inference time of the model. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0034] Figure 2 This is a schematic flowchart of a defect detection method according to an embodiment of this application;

[0035] Figure 3 This is a specific example of the regions of a product image according to an embodiment of this application;

[0036] Figure 4 This is an example of a semantic segmentation model involved in the embodiments of this application;

[0037] Figure 5A This is a specific example of the region segmentation result according to an embodiment of this application;

[0038] Figure 5B This is another specific example of the region segmentation result according to the embodiments of this application;

[0039] Figure 6 This is a specific example of obtaining an image of the region to be tested according to an embodiment of this application;

[0040] Figure 7 This is an example of a target detection model involved in the embodiments of this application;

[0041] Figure 8 This is a schematic flowchart of another defect detection method according to an embodiment of this application;

[0042] Figure 9 This is a schematic flowchart of another defect detection method according to an embodiment of this application;

[0043] Figure 10A This is an example of the defect detection results and region segmentation results output by the model according to an embodiment of this application;

[0044] Figure 10B This is an example of a post-processed defect detection result according to an embodiment of this application;

[0045] Figure 11A This is another example of the defect detection results and region segmentation results output by the model according to the embodiments of this application;

[0046] Figure 11B This is another example of a post-processed defect detection result according to an embodiment of this application;

[0047] Figure 12 A schematic flowchart for defect detection provided in this application;

[0048] Figure 13 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0049] Figure 14 This is a schematic block diagram of a defect detection apparatus according to an embodiment of this application;

[0050] Figure 15 This is a schematic diagram of a model training apparatus according to an embodiment of this application;

[0051] Figure 16 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0053] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0054] In the description of this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. Additionally, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0055] It should also be understood that the descriptions of "first", "second", etc. appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and there is no order to them. They do not indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0056] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0057] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0058] The embodiments of this application are applied to the field of artificial intelligence technology.

[0059] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0060] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0061] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0062] This application's embodiments may relate to Computer Vision (CV) technology within artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0063] This application's embodiments may also relate to Machine Learning (ML) in artificial intelligence technology. ML is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0064] Deep learning (DL) is a branch of machine learning that attempts to perform high-level abstractions of data using multiple processing layers containing complex structures or multiple nonlinear transformations. Deep learning learns the inherent patterns and hierarchical representations of training sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm, and its performance in speech and image recognition far surpasses previous related technologies.

[0065] The following describes the relevant terms used in this application.

[0066] Object detection is an important field within computer vision. Input images for object detection often contain many objects, and the goal of object detection is to determine the location of each object and its category. For example, object detection can output the bounding box of the object and its category. Optionally, object detection can also output the confidence score for the category.

[0067] The bounding box can be a rectangle that can just contain the object, and it can be determined by the coordinates of the relevant position (such as the center point or vertex) within the rectangle.

[0068] Defect detection is a type of target inspection technology. Its target is defects on the appearance of a product, and the output can include a defect bounding box and a defect category. For example, the product can be an industrial product, such as an industrial device or part, and the defects can include foreign objects, dents, scratches, wrinkles, corrosion, dents, dirt, etc., without limitation.

[0069] Image segmentation is another important field in computer vision. It involves dividing an image into several specific regions with unique properties and identifying the target of interest. Image segmentation is a pixel-level classification process, capable of determining which category a pixel in an input image belongs to.

[0070] Region segmentation is a type of image segmentation that uses the uniformity of the same region within the same image to identify different regions. In this embodiment, region segmentation refers to performing multi-class deep learning semantic segmentation on an image to locate different regions within the image.

[0071] Invalid area: Areas in the image that do not belong to the product image, such as black background images.

[0072] Interference area: Areas that belong to the product image but do not need to be inspected for defects, such as the internal cavity of industrial parts.

[0073] Area to be tested: This refers to the area within the product image that needs to be inspected for defects. The area to be tested can be further divided into at least two categories, such as Category A and Category B, or others.

[0074] Optionally, different categories of test areas may have different sensitivities to defects. For example, in a category A test area, a scratch may not be considered a true defect, while in a category B test area, a scratch is considered a true defect.

[0075] Figure 1 This is a schematic diagram of an application scenario involved in an embodiment of this application.

[0076] like Figure 1As shown, the system includes an acquisition device 101, a computing device 102, and a display device 103. The acquisition device 101 is used to acquire images and obtain images to be inspected. The computing device 102 is used to process the images acquired by the acquisition device 101, such as performing defect detection. The display device 103 is used to display the defect detection results obtained by the computing device 102, which may include defect bounding boxes and defect types.

[0077] For example, the acquisition device 101 can be an image acquisition device such as a camera or webcam, without limitation.

[0078] For example, the computing device 102 can be a server. There can be one or more servers. When there are multiple servers, at least two servers provide different services, and / or at least two servers provide the same service, such as providing the same service in a load-balanced manner; this embodiment of the application does not limit this. The server may contain a neural network model, which supports the training and application of the neural network model. The server may also contain a defect detection device for detecting defects in product images, and the server supports the application of the defect detection device.

[0079] The aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Servers can also become nodes in a blockchain.

[0080] For example, computing device 101 may also be a user device, such as a mobile phone, tablet computer, laptop computer, handheld computer, mobile internet device (MID) or other terminal device with data processing capabilities.

[0081] In some embodiments, when the computing device 102 has a display function, the display device 103 can be a display in the computing device 102.

[0082] In some embodiments, the display device 103 and the computing device 102 are different devices, and the display device 103 is connected to the computing device 102 via a network. The network can be an intranet, the Internet, Global System for Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, a voice communication network, or other wireless or wired networks.

[0083] In existing defect detection technologies, a full-area product image is input into a machine learning-based defect detection model for full-area defect detection. During training, the model labels defects in the target area of ​​the product, omitting defects in other parts of the product. However, because the model can only distinguish whether something is a defect, it cannot differentiate the location of the defect, such as whether it's in the target area or a distracting region. This can lead to false positives (false detections) outside the target area during testing. Furthermore, whether the detected defect is located in the target area or a distracting region is crucial for determining its authenticity. Existing defect detection models do not consider the defect's location, failing to meet the practical judgment rules for defect detection in industrial applications.

[0084] Defect detection based on full-area product images is a method for directly detecting product defects, but it has the following drawbacks:

[0085] 1) Defects in different products and different areas have very similar characteristics, which often makes it impossible for defect detection modules to distinguish them, or causes unnecessary over-detection:

[0086] 2) Defects often account for a very small proportion of the entire image. In this case, defect detection is a small target detection. The defect features are not obvious, which increases the difficulty of model detection.

[0087] 3) Defect detection across the entire image requires computation on the entire image, which is very time-consuming, easily leads to a waste of hardware resources, and increases the training and inference time of the model.

[0088] To address the aforementioned technical issues, embodiments of this application input the image to be detected into a region segmentation model to obtain a region segmentation result for the image, which includes a region to be detected. The region to be detected includes areas in the product image that need to be detected for defects, but excludes areas in the product image that do not need to be detected for defects. Then, an image of the region to be detected is obtained based on the region to be detected, and the image of the region to be detected is input into a defect detection model to obtain a first defect detection result.

[0089] Therefore, this embodiment of the application obtains the test region of the image to be detected by performing region segmentation. Since the test region includes areas in the product image that need to be detected for defects, and excludes areas in the product image that do not need to be detected for defects, the test region image obtained based on the test region is input into the defect detection model. The resulting defect detection result includes the defect bounding box and defect type of the test region, which helps to distinguish defects in the product's test region and thus reduces false detections outside the test region output by the model. Because this embodiment of the application considers the location of the defect, it conforms to the judgment rules for defect detection in industrial applications, which helps to improve the reliability of defect detection.

[0090] Optionally, embodiments of this application may further determine the position of at least one defect bounding box in the first defect detection result in the product image based on the first defect detection result and the region segmentation result, and further determine the second defect detection result of the image to be detected based on the position of the at least one defect bounding box in the product image.

[0091] In this way, the embodiments of this application can further determine the position of the defect bounding box in the defect detection result output by the defect identification model, such as whether it is in the area to be tested, or in which type of area to be tested, so as to more accurately distinguish the defects in the product's area to be tested based on the position of the bounding box, and further improve the reliability of defect detection.

[0092] Furthermore, in this embodiment, the image of the region to be tested is obtained based on the region to be tested, and the image of the region to be tested is input into the defect detection model, which can help increase the proportion of defects in the whole image and amplify the defect features, thereby helping to improve the accuracy of defect identification.

[0093] Furthermore, in the defect detection embodiment of this application, only the image of the region to be detected needs to be processed, instead of the entire image, which helps to reduce the waste of hardware resources and reduce the training and inference time of the model.

[0094] This application embodiment can be used in product appearance quality inspection scenarios. For example, the product can be an industrial product, such as industrial parts or components, without limitation. Specifically, a camera can be used to photograph the product (e.g., the whole or a part) to obtain an image to be inspected. This image includes not only the area to be inspected but may also contain invalid and interfering areas (such as the internal cavity of an industrial part). Then, the image to be inspected can be input into a region segmentation model to obtain a region detection result, including the area to be inspected. After obtaining the image of the area to be inspected based on the area to be inspected, the image is input into a defect detection model to obtain a first defect detection result. This helps to detect defects within the area to be inspected of the product during defect detection, while invalid and interfering areas are not within the scope of defect detection. This facilitates the differentiation of defects within the area to be inspected, thereby reducing false detections outside the area to be inspected by the model output. Furthermore, this application embodiment considers the location of defects, conforming to the judgment rules for defect detection in industrial applications, which helps to improve the reliability of defect detection.

[0095] Optionally, based on the first defect detection result and the region segmentation result, the position of at least one defect bounding box in the first defect detection result within the product image can be determined, such as whether it is within the area to be tested, or which specific type of area to be tested it belongs to. Further, based on the position of this at least one defect bounding box in the product image, a second defect detection result for the image to be tested can be determined, for example, by filtering out interference areas or by using different judgment methods to identify true defects for different types of areas to be tested. This allows for more accurate differentiation of defects in the product's area to be tested, further improving the reliability of defect detection.

[0096] The embodiments of this application not only improve the reliability of defect detection but also accelerate model inference speed, making them adaptable to product appearance quality inspection under various complex scenarios (such as different lighting and posture changes). Furthermore, the solutions of this application can be easily extended to other projects, exhibiting good reproducibility.

[0097] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0098] Figure 2 A schematic flowchart of a defect detection method 200 according to an embodiment of this application is shown. Method 200 can be executed by any electronic device with data processing capabilities; for example, the electronic device may be implemented as… Figure 1 The computing device 102 in the application is not limited in this respect.

[0099] like Figure 2 As shown, method 200 includes steps 210 to 240.

[0100] Step 210: Obtain the image to be detected, which includes a product image.

[0101] For example, the camera can take pictures of a part or the whole of the product to obtain an image to be inspected, including an image of the product. The product can be an industrial product with defects to be inspected, such as an industrial part or component, without limitation.

[0102] Step 220: Input the image to be detected into the region segmentation model to obtain the region segmentation result. The region segmentation result includes the region to be tested, which includes the area in the product image that needs to be detected for defects, but does not include the area in the product image that does not need to be detected for defects, i.e. the area that needs to be inspected for appearance.

[0103] Specifically, this region segmentation model can locate and classify regions of a product image based on semantic segmentation using deep learning. In some embodiments, the regions of the product image may include regions to be tested, invalid regions, and interference regions. Regions to be tested can be found in the description above. Interference regions include areas in the product image that do not require defect detection, i.e., areas outside the regions to be tested that still belong to the product, such as internal cavities of parts. Invalid regions do not belong to the product, such as background images.

[0104] Optionally, the test area can be further subdivided into at least two categories of test areas, such as category A test areas and category B test areas, or other types of test areas, without limitation.

[0105] Figure 3 This example illustrates various regions of a product image, where black represents invalid regions, i.e., the background image. The regions to be tested can further include Class A and Class B regions, and other areas of the product image that do not require quality inspection are considered interference regions.

[0106] The region segmentation model is obtained by training a neural network model based on product sample images.

[0107] In some embodiments, the product sample image can be labeled (e.g., manually), for example, by labeling the region to be tested. Optionally, different categories of regions to be tested can also be labeled, for example, there can be n types of regions to be tested, where n is a positive integer. Each category of region to be tested corresponds to a class in the region segmentation model.

[0108] Optionally, interfering or invalid areas in the product sample image can be marked. Alternatively, interfering and invalid areas can be treated as background areas.

[0109] As an example, when n types of test regions are labeled, and invalid and interference regions are uniformly labeled as background regions, the total number of categories corresponding to the region segmentation model is n+1.

[0110] In some embodiments, the region segmentation model includes the deep learning-based SegFormer semantic segmentation model. For example... Figure 4 As shown, the SegFormer semantic segmentation model 300 can include two main parts: 1) a hierarchical Transformer encoder 310, used to generate high-resolution coarse features and low-resolution fine features; 2) a lightweight Multilayer Perceptron (MLP) decoder 320, used to fuse these multi-level features to generate the final semantic segmentation mask and predict the type of each region. The pyramid concept is used in the encoding process to fuse features at multiple levels of different resolutions.

[0111] In some embodiments, the SegFormer semantic segmentation model can be lightweighted, for example by simplifying its convolutional layers and reducing the number of convolutional layers, thereby improving the model's training efficiency and region classification efficiency without significantly sacrificing the model's detection accuracy.

[0112] In some embodiments, the entire image can be input into the model during training and testing of the region segmentation model.

[0113] In other embodiments, images can be resized before being input into the model during training and testing of the region segmentation model. For example, a 2048×2048 resolution image can be resized to a 1024×1024 resolution image to reduce the image resolution and thus reduce the time required for the region segmentation model.

[0114] In some embodiments, product sample images may not be labeled, and the region segmentation model may be trained using an unsupervised training method. This application does not limit this.

[0115] After the region segmentation model is trained, the image to be detected can be input into the model, and the region segmentation result will be output. Figure 5A and Figure 5B Specific examples of region segmentation results are shown. For example... Figure 5A As shown, the region segmentation result includes the Class A test region (the grayscale part on the left side of the figure), which can effectively filter out the interference region (the light-colored part on the right side of the Class A test region). Figure 5B As shown, the region segmentation results include Class A and Class B test regions, which can further subdivide the test regions.

[0116] Step 230: Obtain an image of the region to be tested, based on the region to be tested. This image includes the region to be tested.

[0117] In some embodiments, the minimum bounding rectangle of the region to be tested can be obtained, and the minimum bounding rectangle can be extended outward by a first pixel to obtain an image of the region to be tested.

[0118] like Figure 6 As shown, by extending the minimum bounding rectangle 601 of the area to be tested outward by one pixel, for example by extending it upward, downward, left, and right by at least one row of pixels, a rectangular area 602 can be obtained. The area corresponding to the rectangular area 602 is the image of the area to be tested mentioned above.

[0119] In some embodiments, the image of the region to be tested may also be referred to as the region of interest (ROI) or a dynamic ROI, which is not limited in this application.

[0120] In other embodiments, the region image to be tested (or ROI, dynamic ROI) may also be the region to be processed that is outlined by a circle, ellipse, irregular polygon, etc., and this application does not limit it in this way.

[0121] In some embodiments, the image to be detected can be cropped online based on the region to be detected to obtain an image of the region to be detected. For example, the image to be detected can be cropped online using a dynamic region of interest (ROI) as the boundary to obtain a sub-image of the ROI.

[0122] In this embodiment, the image size of the region to be tested is significantly reduced compared to the original image to be detected. Consequently, the proportion of the product and its defects in the image increases, which is beneficial for accelerating model training and inference. Furthermore, as the proportion of defects in the image is magnified, the features of the defects are also magnified, making it more advantageous for detecting subtle defects in the product.

[0123] Step 240: Input the image of the region to be tested into the defect detection model to obtain a first defect detection result. The first defect detection result includes at least one defect bounding box and the defect type of the defect bounding box.

[0124] In some embodiments, the defect detection model can be a deep learning-based target detection model, such as Faster R-CNN, and this application does not limit it.

[0125] For example, such as Figure 7As shown, the defect detection model 700 may include a backbone network 710, a neck network 720, and a detection head 730. The backbone network 710 is used for feature extraction from the image of the region to be tested, such as using ResNet50 or Swin-Transformer. The neck network 720 is used to perform multi-scale fusion and enhancement of the extracted features to obtain better features; a feature pyramid network (FPN) is typically used. The detection head 730 is used for class prediction and bounding box regression, outputting the defect detection results.

[0126] The defect detection results include at least one defect bounding box and the defect type corresponding to each defect bounding box. Optionally, the defect detection results may also include the confidence level that the defect bounding box belongs to the corresponding defect type.

[0127] In some embodiments, the defect detection model can be obtained by training a neural network model based on product sample images, and this application does not limit this.

[0128] Therefore, this embodiment of the application obtains the test region of the image to be detected by performing region segmentation. Since the test region includes areas in the product image that need to be detected for defects, and excludes areas in the product image that do not need to be detected for defects, the test region image obtained based on the test region is input into the defect detection model. The resulting defect detection result includes the defect bounding box and defect type of the test region, which helps to distinguish defects in the product's test region and thus reduces false detections outside the test region output by the model. Because this embodiment of the application considers the location of the defect, it conforms to the judgment rules for defect detection in industrial applications, which helps to improve the reliability of defect detection.

[0129] Furthermore, in this embodiment, the image of the region to be tested is obtained based on the region to be tested, and the image of the region to be tested is input into the defect detection model, which can help increase the proportion of defects in the whole image and amplify the defect features, thereby helping to improve the accuracy of defect identification.

[0130] Furthermore, in the defect detection embodiment of this application, only the image of the region to be detected needs to be processed, instead of the entire image, which helps to reduce the waste of hardware resources and reduce the training and inference time of the model.

[0131] In some embodiments, such as Figure 8 As shown, method 200 may further include steps 250 and 260.

[0132] Step 250: Based on the first defect detection result and the region segmentation result, determine the position of at least one defect bounding box in the product image.

[0133] Specifically, the region segmentation results can distinguish between the test area and the interference area in the product image. For each defect bounding box output by the defect detection model, the region segmentation results can be used to determine whether it is in the test area or what type of test area it is in.

[0134] In some embodiments, see Figure 9 The location of the defect bounding box in the product image can be determined through steps 251 to 253.

[0135] 251. Obtain the first matrix of the image size of the region to be tested. In this first matrix, the values ​​inside each defect bounding box are non-zero, and the values ​​outside each defect bounding box are zero.

[0136] For example, for each defect bounding box, a matrix A of dynamic ROI size (an example of the first matrix) can be initialized, in which the values ​​corresponding to the inside of the defect bounding box are set to 1, and the values ​​corresponding to the outside of the defect bounding box are set to 0.

[0137] 252. Obtain the second matrix of the image size of the region to be tested. The values ​​in the second matrix corresponding to the region to be tested are non-zero, and the values ​​corresponding to the region outside the region to be tested are 0.

[0138] For example, for each defect bounding box, a region segmentation matrix B of dynamic ROI size (an example of a second matrix) is initialized simultaneously. The values ​​of the background region in matrix B are set to 0, and the values ​​of the region to be tested are set to non-zero.

[0139] In some embodiments, when the region segmentation result contains a region of a certain type, the value of that region can be set to 1.

[0140] In some embodiments, when the region segmentation result includes at least two types of test regions, the at least two types of test regions correspond to different non-zero values ​​in the second matrix. For example, in the matrix B above, the value corresponding to the test region of type A is set to 1, the value corresponding to the test region of type B is set to 2, and so on.

[0141] 253. Multiply the first matrix and the second matrix to obtain the third matrix. In the third matrix, the first defect bounding box corresponding to the non-zero value is located in the area to be tested, and the second defect bounding box corresponding to the zero value is located in the area outside the area to be tested.

[0142] In other words, the first defect bounding box is the defect bounding box located in the region to be tested, and the second defect bounding box is the defect bounding box located in the region outside the region to be tested (i.e., the interference region).

[0143] For example, multiplying the above matrices A and B yields matrix C (an example of a third matrix).

[0144] When the region segmentation result contains a test region of one category, the first defect bounding box corresponding to the value 1 in matrix C is located in the test region, and the second defect bounding box corresponding to the value 0 is located in the background region (i.e., the interference region).

[0145] When the region segmentation result contains at least two types of test regions, the first defect bounding box in the third matrix corresponding to different non-zero values ​​in different types of test regions. For example, in matrix C, the first defect bounding box in type A test region corresponds to the value 1, the first defect bounding box in type B test region corresponds to the value 2, and so on.

[0146] See Figure 10A (a) Figure shows the defect detection results output by the defect detection model, and (b) Figure shows the region segmentation results, including two Class A test regions, one Class B test region, and background regions (invalid regions and interference regions). Optionally, according to steps 251 to 253 above, it can be determined that: the defect bounding box (YS|0.9) is located on the left Class A test region, the defect bounding box (ZW|0.7) is located on the right Class A test region, the defect bounding box (ZW|0.8) is located on the Class B test region, and the defect bounding boxes (CS|0.8) and (YS|0.1) are located on the interference region.

[0147] In the defect enclosing box, the letter before the vertical line indicates the defect type, where CS represents scratch, YS represents dent, and ZW represents dirt. The number after the vertical line indicates the confidence level that the corresponding defect enclosing box belongs to the defect type in parentheses.

[0148] See Figure 11A (a) Figure shows the defect detection results output by the defect detection model, and (b) Figure shows the region segmentation results, including two Class A test regions, one Class B test region, and a background region (invalid region). Optionally, according to steps 251 to 253 above, it can be determined that: the defect bounding box (YS|0.8) is located on the Class B test region, and the defect bounding box (ZW|0.7) is located on the right-hand Class A test region.

[0149] Step 260: Determine a second defect detection result for the image to be detected based on the position of at least one defect bounding box in the product image. The second defect detection result includes at least a portion of the first defect detection result.

[0150] Therefore, the embodiments of this application can further determine the position of the defect bounding box in the defect detection result output by the defect identification model, such as whether it is in the area to be tested, or in which type of area to be tested, so as to more accurately distinguish the defects in the area to be tested of the product based on the position of the bounding box, and further improve the reliability of defect detection.

[0151] Specifically, not all defect bounding boxes detected by the defect detection model are true defects. Therefore, the defect detection results output by the model can be further post-processed, and combined with region segmentation results to further determine whether the defect bounding boxes are true defects.

[0152] In some embodiments, the second defect detection result can be determined based on the first defect bounding box.

[0153] As one possible implementation, the second defect detection result may include the first defect bounding box mentioned above, that is, defects in the area to be tested can be regarded as true defects, while defects outside the area to be tested (i.e., the interference area) (i.e., the second defect bounding box) can be filtered out.

[0154] See Figure 10B ,for Figure 10A In the case described above, under this implementation, the second defect detection result includes the defect bounding box (YS|0.9) on the left A-class test area, the defect bounding box (ZW|0.7) on the right A-class test area, and the defect bounding box (ZW|0.8) on the B-class test area, while the defect bounding boxes (CS|0.8) and (YS|0.1) located in the interference area are filtered out.

[0155] As another possible implementation, when the first defect bounding box is determined to be a first defect type within the first region to be tested, the second defect detection result is determined to not include the first defect bounding box. Here, the first region to be tested is one of the aforementioned at least two types of regions to be tested.

[0156] For example, an indentation (YS) appearing in a Class B test area is not considered a true defect. In this case, when the first defect bounding box is determined to be YS in the Class B test area, the defect is not considered a true defect, meaning that the final second defect detection result does not include this defect.

[0157] For a concrete example, see Figure 11B ,for Figure 11A In the case described above, under this implementation, the second defect detection result includes the defect bounding box (ZW|0.7) on the right-hand A-class test area, but excludes the defect bounding box (YS|0.8) on the B-class test area.

[0158] As another possible implementation, when the confidence level of the defect type corresponding to the first defect bounding box is greater than or equal to a confidence threshold, the second defect detection result is determined to include the first defect bounding box. The confidence threshold for the same defect type varies across different categories of test regions.

[0159] Specifically, different confidence thresholds can be set for different defect types in different categories of test areas. Only when the corresponding defect type in the test area exceeds the corresponding confidence threshold is the defect judged as a true defect.

[0160] For example, the confidence threshold is set to 0.5 for ZW in the test region of class A, and to 0.9 for ZW in the test region of class B.

[0161] As a concrete example, see Figure 10B At this point, the confidence level of the defect bounding box (ZW|0.8) in the B-type test area is less than the confidence level threshold of ZW in the B-type test area, which is 0.9, so it can be determined that it does not belong to a true defect.

[0162] As another concrete example, see Figure 11B At this point, the confidence level of the defect bounding box (ZW|0.7) in the A-class test area is greater than the confidence level threshold of ZW in the A-class test area (0.5), so it can be determined that it is a true defect.

[0163] In some embodiments, the confidence threshold for the same defect type in different categories of test areas may be the same, and this application does not limit this.

[0164] As another possible implementation, when the number of defect types corresponding to the first defect bounding box is greater than or equal to a quantity threshold, the second defect detection result is determined to include the first defect bounding box. The quantity threshold for the same defect type varies across different categories of test areas.

[0165] For example, for ZW defects in type A test regions, the quantity threshold is set to 1, and for ZW defects in type B test regions, the confidence threshold is set to 2. That is, when the number of ZW defects in type A test regions is greater than or equal to 1, these defects are considered true defects; when the number of ZW defects in type B test regions is greater than or equal to 2, these defects are considered true defects; and when the number of ZW defects in type B test regions is 1, the defect is filtered out.

[0166] Therefore, the embodiments of this application can flexibly apply different post-processing strategies to determine whether the defect bounding box belongs to a real defect based on the position of the defect bounding box in the product image, such as whether it is located in the area to be tested, or which type of area to be tested it is located in. This enables more accurate differentiation of defects in the product's area to be tested, and whether the defect is a real defect, thereby further improving the reliability of defect detection.

[0167] Figure 12 This is a schematic flowchart illustrating a defect detection method provided in this application. Figure 12 As shown, the image to be detected can be input into the region segmentation model 810. The region segmentation model 810 can locate and classify regions of the product image based on semantic segmentation using deep learning, outputting region segmentation results, including the region to be detected and the background region (including interference regions and invalid regions). Optionally, the region to be detected can be further subdivided. Then, the region segmentation results can be input into the dynamic ROI module 820 to obtain the ROI sub-image corresponding to the region to be detected (i.e., an example of the aforementioned region to be detected image). Afterwards, the ROI sub-image is input into the defect detection model 830, outputting the model detection results (i.e., an example of the aforementioned first defect detection results). Finally, the model detection results and the region segmentation results can be output together to the post-processing module 840. The post-processing model 840 combines the region segmentation results to perform post-processing on the model detection results output by the defect detection model, such as filtering out the detection results of interference regions, or using different judgment methods to identify true defects for different categories of regions to be detected.

[0168] Figure 13 A schematic flowchart of a model training method 900 provided in an embodiment of this application is shown. Method 900 can be executed by any electronic device with data processing capabilities; for example, the electronic device can be implemented as... Figure 1 The computing device 102 in the application is not limited in this respect.

[0169] like Figure 13 As shown, method 200 includes steps 910 to 940.

[0170] Step 910: Obtain a training sample set, which includes sample images of the product and annotation information of the test area of ​​the sample images. The test area includes the area of ​​the product image that needs to be detected for defects, but does not include the area of ​​the product image that does not need to be detected for defects.

[0171] Step 920: Input the sample image into the region segmentation model to obtain the predicted region segmentation result, which includes the predicted region to be tested.

[0172] For details on the region segmentation model, please refer to the relevant description above, which will not be repeated here.

[0173] Step 930: Determine the model loss based on the predicted region segmentation results and the annotation information of the test region in the sample image.

[0174] Step 940: Update the parameters of the region segmentation model based on the model loss to obtain the trained model.

[0175] Therefore, based on the sample image of the product and the annotation information of the region to be tested in the sample image, the embodiments of this application can train a region segmentation model, thereby using the region segmentation model to perform region segmentation on the entire image of the product and obtain a region segmentation result containing the region to be tested. Furthermore, by using the region segmentation model to perform region segmentation on the image to be detected, and inputting the image of the region to be tested obtained based on the region segmentation result into the defect detection model, it is beneficial to distinguish defects in the region to be tested of the product.

[0176] Furthermore, in this embodiment of the application, the region segmentation model is used to segment the image to be detected, thereby increasing the proportion of defects in the image and amplifying the features of defects, thereby improving the accuracy of the defect detection model and accelerating the inference speed.

[0177] In some embodiments, the training sample set further includes annotation information of background regions, which include invalid regions and interference regions; wherein, invalid regions include background images, and interference regions include areas in product images that do not need to be detected for defects.

[0178] In some embodiments, the training sample set of the region segmentation model may contain a small amount of labeled data, and this application does not limit this.

[0179] In some embodiments, the image of the region to be tested can be determined based on the predicted region segmentation result output by the region segmentation model, and the image of the region to be tested can be used as a sample to train the defect segmentation model to obtain the trained defect detection model, which is beneficial to the convergence of the model.

[0180] The specific embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0181] It should also be understood that, in the various method embodiments of this application, the sequence numbers of the above processes do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that these sequence numbers can be interchanged where appropriate so that the embodiments of this application described can be implemented in a sequence other than those illustrated or described.

[0182] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figures 14 to 16 The following describes in detail the device embodiments of this application.

[0183] Figure 14 This is a schematic block diagram of a defect detection apparatus 10 according to an embodiment of this application. Figure 14 As shown, the device 10 may include a first acquisition unit 11, a region segmentation model 12, a second acquisition unit 13, and a defect detection model 14.

[0184] The first acquisition unit 11 is used to acquire an image to be detected, the image to be detected including a product image;

[0185] Region segmentation model 12 is used to input the image to be detected and obtain a region segmentation result, wherein the region segmentation result includes the region to be detected; wherein the region to be detected includes the region in the product image that needs to be detected for defects, but does not include the region in the product image that does not need to be detected for defects;

[0186] The second acquisition unit 13 is used to acquire an image of the region to be tested based on the region to be tested.

[0187] Defect detection model 14 is used to input the image of the area to be tested into the defect detection model to obtain a first defect detection result;

[0188] The region segmentation model and the defect detection model are respectively trained on the neural network model based on the product sample image.

[0189] In some embodiments, the first defect detection result includes at least one defect bounding box and the defect type of the defect bounding box; the device 10 may further include a determining unit 15, configured to:

[0190] Based on the first defect detection result and the region segmentation result, the position of the at least one defect bounding box in the product image is determined;

[0191] The second defect detection result of the image to be detected is determined based on the position of the at least one defect bounding box in the product image.

[0192] In some embodiments, the determining unit 15 is specifically used for:

[0193] Obtain a first matrix of the image size of the region to be tested, wherein the values ​​within each defect bounding box in the first matrix are non-zero, and the values ​​outside each defect bounding box are zero;

[0194] Obtain a second matrix of the image size of the region to be tested, wherein the values ​​in the second matrix corresponding to the region to be tested are non-zero, and the values ​​corresponding to the region outside the region to be tested are zero;

[0195] Multiply the first matrix and the second matrix to obtain a third matrix, wherein the first defect bounding box corresponding to the non-zero value in the third matrix is ​​located in the region to be tested, and the second defect bounding box corresponding to the zero value is located in the region outside the region to be tested.

[0196] The second defect detection result is determined based on the first defect bounding box.

[0197] In some embodiments, the region segmentation result includes at least two types of test regions. Specifically, in the second matrix, the at least two types of test regions correspond to different non-zero values; and in the third matrix, the first defect bounding boxes located within different types of test regions correspond to different non-zero values.

[0198] In some embodiments, the determining unit 15 is specifically used for:

[0199] When the confidence level of the defect type corresponding to the first defect bounding box is greater than or equal to the confidence level threshold, the second defect detection result is determined to include the first defect bounding box; wherein, the confidence level thresholds corresponding to the same defect type in different categories of test areas are different.

[0200] In some embodiments, the determining unit 15 is specifically used for:

[0201] The first defect bounding box is determined to be a first defect type in the first region to be tested; wherein, the first region to be tested is one of the at least two types of regions to be tested;

[0202] It is determined that the second defect detection result does not include the first defect bounding box.

[0203] In some embodiments, the region segmentation result further includes a background region, which includes invalid regions and interference regions; wherein, the invalid region includes the background image, and the interference region includes regions in the product image that do not need to be detected for defects.

[0204] In some embodiments, the second acquisition unit 13 is specifically used for:

[0205] Obtain the minimum bounding rectangle of the region to be tested;

[0206] The minimum bounding rectangle is expanded outward by a first pixel to obtain the image of the region to be tested.

[0207] In some embodiments, the region segmentation model includes the deep learning-based SegFormer semantic segmentation model.

[0208] In some embodiments, the defect detection model includes a deep learning-based target detection model.

[0209] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 14 The apparatus 10 shown can execute the above method 200, and the aforementioned and other operations and / or functions of each module in the apparatus 10 are respectively for implementing the corresponding processes in the above methods 200. For the sake of brevity, they will not be described in detail here.

[0210] Figure 15 This is a schematic block diagram of the model training apparatus 20 according to an embodiment of this application. Figure 15 As shown, the device 20 may include an acquisition unit 21, a region segmentation model 22, a determination unit 23, and an update unit 24.

[0211] The acquisition unit 21 is used to acquire a training sample set, which includes sample images of the product and annotation information of the test area of ​​the sample images. The test area includes the area in the product image that needs to be detected for defects, but does not include the area in the product image that does not need to be detected for defects.

[0212] Region segmentation model 22 is used to input the sample image and obtain a predicted region segmentation result, wherein the predicted region segmentation result includes a predicted region to be tested.

[0213] The determining unit 23 is used to determine the model loss based on the predicted region segmentation result and the annotation information of the test region of the sample image;

[0214] The update unit 24 is used to update the parameters of the region segmentation model according to the model loss to obtain the trained model.

[0215] In some embodiments, the training sample set further includes annotation information of background regions, which include invalid regions and interference regions; wherein, the invalid regions include background images, and the interference regions include areas in the product image that do not need to be detected for defects.

[0216] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 15 The apparatus 20 shown can execute the above method 900, and the aforementioned and other operations and / or functions of each module in the apparatus 20 are respectively for implementing the corresponding processes in the above methods 900. For the sake of brevity, they will not be described in detail here.

[0217] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0218] Figure 16 This is a schematic block diagram of the electronic device 30 provided in the embodiments of this application.

[0219] like Figure 16 As shown, the electronic device 30 may include:

[0220] The system includes a memory 31 and a processor 32. The memory 31 stores computer programs and transfers the program code to the processor 32. In other words, the processor 32 can retrieve and run the computer programs from the memory 31 to implement the methods described in the embodiments of this application.

[0221] For example, the processor 32 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0222] In some embodiments of this application, the processor 32 may include, but is not limited to:

[0223] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0224] In some embodiments of this application, the memory 31 includes, but is not limited to:

[0225] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0226] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0227] like Figure 16 As shown, the electronic device 30 may further include:

[0228] Transceiver 33, which can be connected to processor 32 or memory 31.

[0229] The processor 32 can control the transceiver 33 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 33 may further include antennas, and the number of antennas may be one or more.

[0230] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0231] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0232] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0233] It is understood that specific embodiments of this application may involve user information and other related data. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0234] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0236] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0237] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for defect detection, characterized in that, include: Acquire an image to be detected, the image to be detected including a product image; The image to be detected is input into a region segmentation model to obtain a region segmentation result, which includes the region to be tested; wherein, the region to be tested includes the region in the product image that needs to be detected for defects, but does not include the region in the product image that does not need to be detected for defects. Based on the area to be tested, obtain an image of the area to be tested; The image of the region to be tested is input into the defect detection model to obtain the first defect detection result; The region segmentation model and the defect detection model are respectively trained on the neural network model based on the product sample image; The first defect detection result includes at least one defect bounding box and the defect type of the defect bounding box, and the method further includes: Based on the first defect detection result and the region segmentation result, the position of the at least one defect bounding box in the product image is determined; the position includes whether the defect bounding box is in the region to be tested, or the type of the region to be tested. Based on the position of the at least one defect bounding box in the product image, a second defect detection result of the image to be detected is determined. The second defect detection result includes whether the at least one defect bounding box belongs to a true defect. The confidence threshold for the same defect type is different in different categories of test areas.

2. The method according to claim 1, wherein determining the position of the at least one defect bounding box in the product image based on the first defect detection result and the region segmentation result comprises: Obtain a first matrix of the image size of the region to be tested, wherein the values ​​within each defect bounding box in the first matrix are non-zero, and the values ​​outside each defect bounding box are zero; Obtain a second matrix of the image size of the region to be tested, wherein the values ​​in the second matrix corresponding to the region to be tested are non-zero, and the values ​​corresponding to the region outside the region to be tested are zero; Multiply the first matrix and the second matrix to obtain a third matrix, wherein the first defect bounding box corresponding to the non-zero value in the third matrix is ​​located in the region to be tested, and the second defect bounding box corresponding to the zero value is located in the region outside the region to be tested. The step of determining the second defect detection result of the image to be detected based on the position of the at least one defect bounding box in the product image includes: The second defect detection result is determined based on the first defect bounding box.

3. The method according to claim 2, characterized in that, The region segmentation result includes at least two types of regions to be tested; In the second matrix, at least two types of test regions correspond to different non-zero values; In the third matrix, the first defect bounding box located in different types of test regions corresponds to different non-zero values.

4. The method according to claim 3, characterized in that, Determining the second defect detection result based on the first defect bounding box includes: When the confidence level of the defect type corresponding to the first defect bounding box is greater than or equal to the confidence threshold, the second defect detection result is determined to include the first defect bounding box.

5. The method according to claim 3, characterized in that, Determining the second defect detection result based on the first defect bounding box includes: The first defect bounding box is determined to be a first defect type in the first region to be tested; wherein, the first region to be tested is one of the at least two types of regions to be tested; It is determined that the second defect detection result does not include the first defect bounding box.

6. The method according to any one of claims 1-5, characterized in that, The region segmentation result also includes a background region, which includes invalid regions and interference regions; wherein, the invalid region includes the background image, and the interference region includes areas in the product image that do not need to be detected for defects.

7. The method according to any one of claims 1-5, characterized in that, The step of obtaining the image of the region to be tested based on the region to be tested includes: Obtain the minimum bounding rectangle of the region to be tested; The minimum bounding rectangle is expanded outward by a first pixel to obtain the image of the region to be tested.

8. The method according to any one of claims 1-5, characterized in that, The region segmentation model includes the SegFormer semantic segmentation model based on deep learning.

9. The method according to any one of claims 1-5, characterized in that, The defect detection model includes a deep learning-based target detection model.

10. A defect detection device, characterized in that, include: The first acquisition unit is used to acquire an image to be detected, the image to be detected including a product image; A region segmentation model is used as input to the image to be detected to obtain a region segmentation result, wherein the region segmentation result includes the region to be detected; wherein the region to be detected includes the region in the product image that needs to be detected for defects, but does not include the region in the product image that does not need to be detected for defects; The second acquisition unit is used to acquire an image of the region to be tested based on the region to be tested. A defect detection model is used to input the image of the region to be tested into the defect detection model to obtain a first defect detection result; The region segmentation model and the defect detection model are respectively trained on the neural network model based on the product sample image; The first defect detection result includes at least one defect bounding box and the defect type of the defect bounding box. The device further includes a processing unit for: Based on the first defect detection result and the region segmentation result, the position of the at least one defect bounding box in the product image is determined; the position includes whether the defect bounding box is located in the region to be tested, or the type of the region to be tested; and Based on the position of the at least one defect bounding box in the product image, a second defect detection result of the image to be detected is determined. The second defect detection result includes whether the at least one defect bounding box belongs to a true defect. The confidence threshold for the same defect type is different in different categories of test areas.

11. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-9.

12. A computer storage medium, characterized in that, Used for storing computer programs, said computer programs including methods for performing any one of claims 1-9.

13. A computer program product, characterized in that, It includes computer program code that, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 1-9.

Images