Surface defect detection method and device, computer device and storage medium

Abstract

The application provides a surface defect detection method and device, computer equipment and a storage medium. The method comprises: acquiring a to-be-inspected surface image and a standard surface image of a to-be-inspected product; performing image segmentation on the to-be-inspected surface image and the standard surface image to acquire at least one to-be-inspected image block corresponding to the to-be-inspected surface image and at least one standard image block corresponding to the standard surface image; inputting the to-be-inspected image block and the standard image block into a pre-trained defect segmentation model to acquire at least one defect segmentation image block; each defect segmentation image block is used to represent pixel difference information between a to-be-inspected image block and a standard image block; and based on the defect segmentation image block, it is determined whether the to-be-inspected product has a surface defect. In this way, the trained defect segmentation model is used for defect segmentation to determine whether the to-be-inspected product has a surface defect, thereby improving the detection efficiency and the accuracy of the detection result of the surface defect detection.

Description

Technical Field

[0001] This application relates to the field of printed matter inspection technology, and in particular to a surface defect detection method, apparatus, computer equipment, and storage medium. Background Technology

[0002] As users and manufacturers of industrial products place increasingly higher demands on product quality, products must not only meet performance requirements but also possess excellent surface quality. However, surface defects are often unavoidable during the manufacturing process, necessitating surface defect detection. Surface defect detection involves inspecting the product surface for defects such as scratches, foreign object obstructions, wrinkles, blemishes, color contamination, and holes, thereby obtaining information about the type, outline, location, and size of these defects.

[0003] In the inspection of surface defects in printed materials, manual inspection is commonly used. This method involves randomly selecting items from a batch of printed materials according to a pre-set sampling rate, and then manually observing the surface of the selected items for defects. However, manual inspection suffers from problems such as low accuracy, poor real-time performance, low efficiency, high labor intensity, and the results being significantly influenced by human experience and subjective factors. Summary of the Invention

[0004] This application provides a surface defect detection method, apparatus, computer equipment, and storage medium, which can improve the detection efficiency of product surface defects.

[0005] In a first aspect, this application provides a surface defect detection method, comprising:

[0006] Acquire images of the surface to be inspected and the standard surface of the product to be inspected;

[0007] Image segmentation is performed on the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one image block to be standard corresponding to the standard surface image; the number of image blocks to be inspected and the number of image blocks to be standard are the same, and the image blocks to be inspected and the image blocks to be standard are also located in the same image region in the corresponding surface image;

[0008] The image block to be inspected and the standard image block are input into a pre-trained defect segmentation model to obtain at least one defect segmentation image block; each defect segmentation image block is used to represent the pixel difference information between an image block to be inspected and a standard image block.

[0009] Based on defect segmentation image blocks, determine whether the product under inspection has surface defects.

[0010] In one possible implementation, image segmentation is performed on the surface image to be inspected and the standard surface image to obtain at least one image block corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image, including:

[0011] The pixel values ​​of pixels at the same position in the image of the surface to be inspected and the standard surface image are subtracted to obtain the intermediate surface image;

[0012] Perform connected component analysis on the intermediate surface image to obtain at least one candidate defect feature map;

[0013] Based on the location information of the corresponding region in the intermediate surface image according to the candidate defect feature map, the surface image to be inspected and the standard surface image are segmented to obtain the image block to be inspected and the standard image block.

[0014] In one possible implementation, determining whether a product under inspection has surface defects based on defect segmentation image patches includes:

[0015] Obtain pixel intersection information between defect segmentation image blocks and candidate defect feature maps;

[0016] If the pixel intersection information is empty, it is determined that there are no surface defects on the surface of the product to be inspected.

[0017] If the pixel intersection information is not empty, it is determined that there is a surface defect on the surface of the product to be inspected.

[0018] In one possible implementation, the defect segmentation model includes an encoder and a decoder;

[0019] The image patch to be inspected and the standard image patch are then input into a pre-trained defect segmentation model to obtain at least one defect segmentation image patch, including:

[0020] The image block to be inspected and the standard image block are input into the encoder. The encoder obtains the first multi-scale feature map of the image block to be inspected at multiple preset resolutions and the second multi-scale feature map of the standard image block at multiple resolutions.

[0021] The first multi-scale feature map and the second multi-scale feature map are input into the decoder. The decoder performs feature fusion processing on the first multi-scale feature map and the second multi-scale feature map and outputs defect segmentation image blocks.

[0022] In one possible implementation, acquiring the image of the surface to be inspected and a standard surface image of the product to be inspected includes:

[0023] Acquire the initial image of the product to be inspected and the initial standard image of the product to be inspected;

[0024] Based on the initial standard image, image registration processing is performed on the initial image to be inspected to obtain the intermediate image to be inspected.

[0025] Image preprocessing is performed on the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image.

[0026] In one possible implementation, image preprocessing is performed on the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image, including:

[0027] Edge detection is performed on the intermediate image to be inspected and the initial standard image to obtain the first edge information of the intermediate image to be inspected and the second edge information of the initial standard image.

[0028] Based on the first edge information, edge dilation processing is performed on the middle image to be inspected to obtain the surface image to be inspected.

[0029] Based on the second edge information, the initial standard image is subjected to edge dilation processing to obtain a standard surface image.

[0030] In one possible implementation, the training process of the defect segmentation model includes:

[0031] Acquire training sample images of the inspected products; the training sample images include a standard sample image and multiple defect sample images, each sample image carries a sample label, the sample label is used to indicate whether the sample image has surface defects;

[0032] Based on the training sample images, generate multiple training sample pairs; each training sample pair includes two sample images.

[0033] Multiple training sample pairs are input into the initial defect segmentation model to be trained. Based on the output of the initial defect segmentation model, the training loss value of the target loss function of the initial defect segmentation model is calculated.

[0034] If the training loss value does not meet the preset convergence condition, the parameters of the initial defect segmentation model are adjusted, and multiple training sample pairs are input into the adjusted initial defect segmentation model again until the training loss value meets the preset convergence condition. Then the training ends and a trained defect segmentation model is obtained.

[0035] In one possible implementation, multiple training samples are generated based on the training sample images, including:

[0036] The standard sample image is combined with multiple defect sample images in sequence to obtain multiple initial sample pairs;

[0037] Multiple initial sample pairs are randomly subjected to data augmentation to obtain multiple training sample pairs;

[0038] Data augmentation processing includes at least one of the following:

[0039] The two sample images in the initial sample pair are randomly swapped to obtain multiple first sample pairs;

[0040] According to the preset pixel offset range, pixel offset processing is performed on any one of the sample images in the initial sample pair to obtain multiple second sample pairs;

[0041] The standard sample image in the initial sample pair is randomly copied as a defective sample image to obtain multiple third sample pairs;

[0042] The pre-extracted image defects are pasted into the standard sample images of the initial sample pairs to obtain multiple fourth sample pairs;

[0043] Multiple training sample pairs include at least one of the following: initial sample pair, first sample pair, second sample pair, third sample pair, and fourth sample pair.

[0044] In one possible implementation, the target loss function of the initial defect segmentation model includes the Dice loss function and the cross-entropy loss function, and the coefficients of the Dice loss function and the cross-entropy loss function are different in the target loss function.

[0045] In one possible implementation, if the product to be inspected is a printed product, then surface defects include shallow adhesive defects.

[0046] Secondly, this application provides a surface defect detection device, comprising:

[0047] The image acquisition module is used to acquire images of the surface to be inspected and the standard surface of the product to be inspected.

[0048] The image segmentation module is used to segment the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image; the number of image blocks to be inspected and standard image blocks are the same, and the image regions of the image blocks to be inspected and standard image blocks in the corresponding surface images are also the same;

[0049] The defect segmentation module is used to input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmented image block; each defect segmented image block is used to represent the pixel difference information between an image block to be inspected and a standard image block;

[0050] The defect detection module is used to determine whether the product under inspection has surface defects based on defect segmentation image blocks.

[0051] Thirdly, this application provides a computer device including a memory and a processor. The memory stores a computer program, and the processor invokes and executes the computer program from the memory to implement the steps of the surface defect detection method shown in any of the first aspects above.

[0052] Fourthly, this application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the surface defect detection method shown in any of the first aspects above.

[0053] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the surface defect detection method shown in any of the first aspects above.

[0054] The technical solution provided in this application can achieve at least the following beneficial effects:

[0055] The surface defect detection method, apparatus, computer equipment, and storage medium provided in this application, for acquiring an image of the surface to be inspected and a standard surface image of a product to be inspected, firstly perform image segmentation on the surface to be inspected and the standard surface image to obtain at least one image block corresponding to the surface to be inspected and at least one image block corresponding to the standard surface image. The number of image blocks to be inspected and the number of image blocks corresponding to the standard image are the same, and the image regions occupied by the image blocks to be inspected and the image blocks corresponding to the standard image are also the same. Then, the image blocks to be inspected and the image blocks corresponding to the standard image are input into a pre-trained defect segmentation model. The defect segmentation model obtains a defect segmented image block corresponding to one image block to be inspected and one image block corresponding to the standard image. This defect segmented image block can reflect the pixel difference information between the image block to be inspected and the standard image block. Finally, based on the defect segmented image block, it is determined whether the product to be inspected has surface defects. Therefore, this application utilizes a pre-trained defect segmentation model for defect segmentation, which is more efficient than manual inspection. Moreover, the defect segmentation model is a dual-input neural network model, with its inputs including a standard image patch corresponding to a standard surface image and a standard image patch corresponding to the surface to be inspected. This addresses the problem of weak generalization ability in deep learning models, eliminating the need for model training for different scenarios and products, thus reducing model training costs. The dual-input defect segmentation model can maintain good defect detection results even when encountering products not present in the training dataset. Furthermore, this application pre-segments the surface image to be inspected and the standard surface image, obtaining at least one image patch corresponding to the surface to be inspected and at least one standard image patch corresponding to the standard surface image. Then, the defect segmentation model processes each image patch, reducing resource consumption during image processing and lowering the computational performance requirements of the computer equipment when segmenting image patches. This allows the surface defect detection method to achieve high-speed real-time computation on low-end computing devices, improving the efficiency of surface defect detection. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application;

[0057] Figure 2 This is a schematic flowchart illustrating a surface defect detection method according to an exemplary embodiment of this application;

[0058] Figure 3 This is a schematic diagram of an image processing flow illustrated in an exemplary embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the structure of a defect detection model shown in an exemplary embodiment of this application;

[0060] Figure 5This is a schematic diagram illustrating the training process of a defect segmentation model according to an exemplary embodiment of this application;

[0061] Figure 6 This is a schematic diagram illustrating a process for generating training sample pairs, as shown in an exemplary embodiment of this application.

[0062] Figure 7 This is a schematic flowchart illustrating another surface defect detection method according to an exemplary embodiment of this application;

[0063] Figure 8 This is a schematic diagram of the structure of a surface defect detection device according to an exemplary embodiment of this application;

[0064] Figure 9 This is a schematic diagram of the structure of another surface defect detection device shown in an exemplary embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0066] Before explaining the surface defect detection method provided in the embodiments of this application, the application scenarios and implementation environment of the embodiments of this application will be introduced first.

[0067] As users and manufacturers of industrial products place increasingly higher demands on product quality, industrial products must not only meet performance requirements but also possess a good appearance, i.e., good surface quality. However, surface defects are often unavoidable during the product manufacturing process. Defects can generally be understood as surface missing parts, surface flaws, or area deviations compared to normal products. Surface defect detection refers to inspecting the surface of a product for scratches, defects, foreign object obstruction, color contamination, holes, and other defects, thereby obtaining information describing the defect type, outline, location, size, and other characteristics.

[0068] Furthermore, surface defects not only affect the aesthetics and comfort of products, but also generally have a negative impact on their performance. Therefore, manufacturers attach great importance to surface defect detection to promptly identify defects and effectively control product quality. Moreover, the results of surface defect detection can be used to analyze technical / operational problems in the production process, thereby eliminating or reducing the generation of defective products, preventing potential trade disputes, and protecting the company's reputation.

[0069] Manual inspection is a traditional method for detecting surface defects in products, but this method has a low sampling rate, low accuracy, poor real-time performance, low efficiency, high labor intensity, and is greatly affected by human experience and subjective factors.

[0070] Furthermore, traditional image processing algorithms suffer from false alarms and missed detections in the detection of shallow adhesive defects in printed materials, requiring repeated manual parameter adjustments. Meanwhile, surface defect detection methods based on single-image input and deep learning suffer from slow inference speed and weak generalization ability; moreover, specific neural network models need to be trained for different scenarios and product types, increasing detection costs.

[0071] While deep learning-based surface defect detection technology can largely overcome the drawbacks of manual inspection, the types of products to be inspected are diverse, and surface defects are becoming increasingly subtle with advancements in manufacturing processes. Therefore, developing a stable and efficient machine vision-based surface defect detection method remains the biggest challenge in the application of machine vision in surface defect detection.

[0072] Based on this, this application provides a surface defect detection method, apparatus, computer equipment, and storage medium. It employs deep learning technology combined with appropriate data augmentation and training strategies to train a defect segmentation model. During actual detection, the defect segmentation model compares and analyzes the pre-processed image block to be inspected with a standard image block to determine whether the product under inspection has surface defects. Thus, the defect segmentation model trained in this application possesses strong stability, generalization ability, and efficiency. Therefore, combining image preprocessing and the defect segmentation model can improve the detection efficiency of surface defects in the product under inspection, and achieve stable detection results across various products and scenarios.

[0073] In an exemplary embodiment, this application can use a computer device to perform image preprocessing on an image of the surface of the product to be inspected and a standard image of the product to be inspected, and analyze the pixel differences between the image of the surface to be inspected and the standard image of the product to be inspected through a pre-trained defect segmentation model, thereby determining whether the product to be inspected has surface defects.

[0074] In one possible implementation, the computer device is structured as follows: Figure 1 As shown, the computer device 100 includes at least one processor 110, a memory 120, a communication bus 130, and at least one communication interface 140.

[0075] The processor 110 can be a general-purpose central processing unit (CPU), network processor (NP), microprocessor, or one or more integrated circuits for implementing the scheme of this application, such as application-specific integrated circuits (AS1C), programmable logic devices (PLDs), or combinations thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0076] Optionally, processor 110 may include one or more CPUs. Computer device 100 may include multiple processors 110. Each of these processors 110 may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU).

[0077] It should be noted that the processor 110 here may refer to one or more devices, circuits and / or processing cores used to process data (such as computer program instructions).

[0078] The memory 120 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions; it may also be a random access memory (ROM) or other type of dynamic storage device capable of storing information and instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0079] Alternatively, the memory 120 may exist independently and be connected to the processor 110 via the communication bus 130; the memory 120 may also be integrated with the processor 110.

[0080] Communication bus 130 is used to transfer information between components (such as between the processor and memory). Communication bus 120 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 1 The diagram uses only one communication bus, but this does not mean that there is only one bus or one type of bus.

[0081] The communication interface 140 is used for the computer device 100 to communicate with other devices or communication networks. The communication interface 140 includes a wired communication interface or a wireless communication interface. The wired communication interface can be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface can be a Wireless Local Area Network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0082] In some embodiments, the computer device 100 may further include output devices and input devices. Figure 1 (Not shown in the image). The output device communicates with the processor 110 and can display information in various ways. For example, the output device can be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 110 and can receive user input in various ways. For example, the input device can be a mouse, keyboard, touch screen device, or sensor device.

[0083] In some embodiments, the memory 120 is used to store a computer program that executes the solution of this application, and the processor 110 can execute the computer program stored in the memory 120. For example, the computer device 100 can call and execute the computer program stored in the memory 120 through the processor 110 to implement the steps of the surface defect detection method provided in the embodiments of this application.

[0084] It should be understood that the surface defect detection method provided in this application can be applied to a surface defect detection device, which can be implemented as part or all of the processor 110 through software, hardware or a combination of software and hardware, and integrated into the computer device 100.

[0085] Next, the technical solutions of this application and how they solve the aforementioned technical problems will be described in detail through embodiments and in conjunction with the accompanying drawings. The embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application.

[0086] In one exemplary embodiment, such as Figure 2 As shown, this application provides a surface defect detection method, which is applied to the above-mentioned... Figure 1 The computer device 100 shown is used as an example. The method may include the following steps:

[0087] Step 210: Obtain the surface image of the product to be inspected and the standard surface image.

[0088] The product to be inspected can be any product produced on an industrial production line, and the surface image to be inspected is a real-time image of the product's surface. Correspondingly, the standard surface image is a surface image of the same type as the product to be inspected, without surface defects.

[0089] As an example, if the product to be inspected is printed material, surface defects include shallow adhesive defects.

[0090] It should be noted that for a batch of products of the same type, during the production process, the corresponding surface images of multiple products to be inspected can be acquired sequentially. For this batch of products, the same standard surface image can be used to perform the surface defect detection method provided in this application, or multiple standard surface images of the products in this batch that do not have surface defects can be used to perform the surface defect detection method provided in this application. This application does not limit this approach.

[0091] It should be understood that the surface of the product under inspection may or may not have defects. If the surface of the product under inspection has defects, then when the image of the surface under inspection and the standard surface image are captured by the same image acquisition device, there will be pixel differences between the image area with defects in the image of the surface under inspection and the corresponding image area in the standard surface image. Based on this, this application analyzes the pixel differences between the image of the surface under inspection and the standard surface image through the following steps 220 and 230.

[0092] As an example, the image acquisition device that captures images of the surface to be inspected and a standard surface can be an RG B camera.

[0093] Since the initial images of the products to be inspected captured in real time on the production line contain redundant information and noise, directly using the initial images as the surface images of the products to be inspected will increase the amount of information processing in the subsequent model processing and may also affect the final defect detection results.

[0094] Therefore, the initial image of the product to be inspected and the initial standard image obtained from actual photography can be preprocessed to obtain the surface image of the product to be inspected and the standard surface image.

[0095] In one possible implementation, such as Figure 3 As shown, the implementation process of step 210 above may include the following sub-steps:

[0096] Step 211: Obtain the initial image of the product to be inspected and the initial standard image.

[0097] Optionally, before performing step 213 below, the initial image to be inspected and the initial standard image can be denoised to remove noise and redundant information from the images.

[0098] The noise reduction process can be achieved through filtering algorithms. These algorithms can include mean filtering, median filtering, Gaussian filtering, etc., and this application does not limit the specific algorithms used.

[0099] Optionally, for ease of calculation, the initial image to be inspected and the initial standard image can be grayscaled to convert the color image captured by the RGB camera into a grayscale image.

[0100] It should be understood that the values ​​of the R, G, and B components in a color image determine a specific pixel, and a pixel can have tens of millions of colors; while a grayscale image is also a type of color image, but its characteristic is that the specific values ​​of the R, G, and B components are consistent, and the pixel value of each pixel in a grayscale image varies from 0 to 255.

[0101] The initial image to be inspected and the initial standard image are converted into corresponding grayscale images. While preserving the image contours and features, the grayscale image can still reflect the contours and textures of the entire image and does not affect subsequent defect detection.

[0102] Step 213: Based on the initial standard image, perform image registration processing on the initial image to be inspected to obtain the intermediate image to be inspected.

[0103] It should be understood that after the initial image to be inspected undergoes image registration processing to obtain the intermediate image to be inspected, there is no need to process the initial standard image in step 213. In other words, after performing step 213, the intermediate image to be inspected and the initial standard image are obtained.

[0104] In actual implementation, due to factors such as shooting angle and lighting, the initial image to be inspected and the initial standard image may not be aligned at the pixel level, resulting in pixel offset between them.

[0105] Therefore, image registration processing is required between the initial image to be inspected and the initial standard image to ensure that the initial image to be inspected and the initial standard image are aligned in terms of image size, pixel position, etc.

[0106] Step 215: Perform image preprocessing on the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image.

[0107] Image preprocessing may include at least one of downsampling, redundant point removal, edge detection, erosion, and dilation.

[0108] In one possible implementation, step 215 is performed as follows: edge detection is performed on the intermediate image to be inspected and the initial standard image to obtain the first edge information of the intermediate image to be inspected and the second edge information of the initial standard image; based on the first edge information, edge dilation processing is performed on the intermediate image to be inspected to obtain the surface image to be inspected; based on the second edge information, edge dilation processing is performed on the initial standard image to obtain the standard surface image.

[0109] Image edges are the most fundamental feature of an image. An edge refers to a discontinuity in the local characteristics of an image; abrupt changes in information such as grayscale or structure are called edges. Examples include abrupt changes in grayscale levels, color, and texture structure.

[0110] Specifically, the edge detection operator used in the edge detection operation can be any one of the following: Roberts operator, Prewitt operator, Sobel operator, Laplacian operator, LoG operator (also known as Marr edge detection operator, or Gaussian-Laplacian operator), or Canny edge detection.

[0111] The Roberts operator, also known as the cross-differentiation algorithm, is a gradient algorithm based on cross-difference, detecting edge lines through local difference calculations. The Roberts operator's template is divided into horizontal and vertical directions, effectively enhancing image edges at ±45 degrees. The Prewitt operator, on the other hand, uses the difference in grayscale values ​​of pixels within a specific region to achieve edge detection. The Prewitt operator uses a template to calculate the pixel values ​​of each pixel within the region, making its edge detection results more pronounced than the Roberts operator in both horizontal and vertical directions. The Laplacian operator is a second-order differential operator in n-dimensional Euclidean space, divided into four-neighbor and eight-neighbor regions. The four-neighbor region calculates the gradient in four directions for the center pixel, while the eight-neighbor region calculates the gradient in eight directions. The LoG operator typically performs Gaussian low-pass filtering before applying the Laplacian operator. The basic idea of ​​Canny edge detection is to first smooth the image using a Gaussian filter, and then use non-extreme suppression techniques to obtain the final edge image.

[0112] As an example, this application embodiment uses the Sobel operator to perform edge detection on the intermediate image to be inspected and the initial standard image to obtain image edge information.

[0113] The Sobel operator adds the concept of weights to the Prewitt operator, arguing that the distance between adjacent pixels has different effects on the current pixel. Pixels that are closer have a greater impact on the current pixel, while pixels that are farther away have a smaller impact, thereby achieving image sharpening and highlighting edge contours.

[0114] Thus, using the Sobel operator can fully consider the mutual influence between pixels, making the calculated edge information more accurate, better describing the true contour in the intermediate image to be inspected, and showing clear contour edges.

[0115] Furthermore, image edge dilation involves adding pixel values ​​to the edges of an image, causing the overall pixel values ​​to expand, thus achieving an image dilation effect. This can also be described as pixel interpolation.

[0116] Based on the positional relationship of adjacent pixels referenced during the dilation process, image dilation includes horizontal dilation, vertical dilation, and omnidirectional dilation.

[0117] As an example, embodiments of this application can perform omnidirectional dilation processing on the edges of the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image.

[0118] It should be understood that omnidirectional dilation involves traversing the target region in the target image, considering only the grayscale values ​​of the target pixel and its four adjacent pixels (top, bottom, left, and right), and checking if there are any intersections with the dilation structuring element, i.e., whether there is at least one corresponding pixel with the same grayscale value. If there is an intersection, the pixel is processed; otherwise, the pixel is deleted.

[0119] Thus, through comprehensive dilation processing, not only can the image boundary points be expanded, but background points that are in contact with the image edges can also be merged into the edge information, so that the dilation processing can better reflect the true boundary between the image edge and the background information, and the dilation processing effect is better.

[0120] Step 220: Perform image segmentation on the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image.

[0121] The number of image blocks to be inspected and the number of standard image blocks are the same, and the image areas of the image blocks to be inspected and the standard image blocks are also the same in the corresponding surface images.

[0122] In other words, in step 220, when performing image segmentation on the surface image to be inspected and the standard surface image, the segmentation is based on the same image position, so that after the segmentation, the position coordinates of the image region corresponding to the image block to be inspected in the surface image to be inspected are the same as the position coordinates of the image region corresponding to the standard image block in the standard surface image; at the same time, the number of image blocks to be inspected and standard image blocks obtained after segmentation are also the same.

[0123] In one possible implementation, step 220 can be implemented as follows: subtracting the pixel values ​​of pixels at the same position in the surface image to be inspected and the standard surface image to obtain an intermediate surface image; performing connected component analysis on the intermediate surface image to obtain at least one candidate defect feature map; and performing image segmentation on the surface image to be inspected and the standard surface image according to the region location information corresponding to the candidate defect feature map in the intermediate surface image to obtain the image block to be inspected and the standard image block.

[0124] Since the image of the surface to be inspected and the standard surface image are image pairs after image registration processing, the pixels at corresponding positions in the two surface images describe the same surface feature of the product to be inspected, so the pixel values ​​can be directly subtracted.

[0125] It should be understood that if both the surface image to be inspected and the standard surface image are grayscale images, then the intermediate surface image obtained after processing will also be a grayscale image. Specifically, the pixel value of each pixel in this intermediate surface image is the absolute value of the difference between the corresponding pixel values ​​in the surface image to be inspected and the standard surface image.

[0126] As an example, at least one candidate defect feature map can be obtained by performing Blob analysis on the intermediate surface image.

[0127] In computer vision, a blob refers to a connected region in an image that has similar colors, textures, and other features. In this embodiment, blob analysis is to analyze the connected regions (called blob blocks) of the same pixels in the intermediate surface image to obtain the feature map of the connected region, i.e., the candidate defect feature map.

[0128] Specifically, Blob analysis involves binarizing the pixel values ​​of each pixel in the intermediate surface image to segment the foreground and background; then performing connected component detection to obtain Blob blocks; and finally extracting pixel features from each Blob block to obtain a candidate defect feature map.

[0129] It should be noted that the number of connected components in the intermediate surface image is the same as the number of candidate defect feature maps. That is, if there is only one connected component in the intermediate surface image, then Blob analysis yields one candidate defect feature map; if there are multiple connected components in the intermediate surface image, then Blob analysis yields candidate defect feature maps corresponding to each connected component.

[0130] Furthermore, taking a candidate defect feature map as an example, the process of performing image segmentation on the surface image to be inspected and the standard surface image based on the candidate defect feature map to obtain the image block to be inspected and the standard image block can be as follows: obtain the region location information corresponding to the candidate defect feature map in the intermediate surface image to obtain the region location information; perform image segmentation on the surface image to be inspected according to the region location information to segment out the local image corresponding to the region location information from the surface image to be inspected to obtain the image block to be inspected; similarly, perform image segmentation on the standard surface image according to the region location information to segment out the local image corresponding to the region location information from the standard surface image to obtain the standard image block.

[0131] Therefore, the embodiments of this application do not directly input the surface image to be inspected and the standard surface image into the trained defect segmentation model. Instead, they combine image preprocessing and image segmentation to reduce the amount of processing information in the image blocks and reduce the computational resource requirements of the defect segmentation model. This enables the surface image-based defect segmentation method to perform high-speed real-time computation on low-end computing devices.

[0132] Step 230: Input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmentation image block.

[0133] Each defect segmentation image block is used to represent the pixel difference information between an image block to be inspected and a standard image block.

[0134] It should be noted that the defect segmentation model provided in this application is a dual-input neural network model. When performing defect segmentation, the model requires two corresponding images as input, and there are no requirements on the input order of the two images.

[0135] Optionally, there are a large number of non-defect features in the candidate defect feature map. After collecting small-sized image blocks at corresponding positions on the surface image to be inspected and the standard surface image based on the regional location information of each candidate defect feature map, the segmented image blocks to be inspected and the standard image blocks can be sorted according to the area size of the candidate defect feature map, so as to ensure that the image blocks to be inspected and the standard image blocks corresponding to the large-area candidate defect feature map can be processed by the defect segmentation model first.

[0136] As an example, taking the shallow adhesive defects in printed materials as an example, in order to improve the detection effect of shallow adhesive defects and achieve better defect detection accuracy, this application uses the SegFormer encoder (a semantic segmentation framework that unifies transformer and lightweight multilayer perceptron decoder) based on the Transformer model (a deep learning model that uses a self-attention mechanism, which can assign different weights according to the different importance of each part of the input data) as the backbone network of the defect segmentation model. Among them, SegFormer uses a hierarchical encoder structure, which can output multi-scale feature maps, and fuse the multi-scale feature maps together in the decoder.

[0137] SegFormer avoids complex decoders, and its proposed MLP decoder can aggregate information from different layers. In other words, SegFormer is similar to the approach in convolutional neural network models that fuse shallow and deep feature maps, aiming to capture both high-resolution coarse-grained features and low-resolution fine-grained features together, thereby optimizing the segmentation results.

[0138] Furthermore, this application's embodiments abandon the Transformer model and the position encoding in the self-attention-based encoder (segmentation transformer, abbreviated as SETR). When the size of the input image is different from the size of the image used during training, it is not necessary to interpolate the position encoding vector (which would lead to performance degradation when the test resolution is different from the training resolution).

[0139] Thus, the simple and effective decoder designed in this application can achieve a defect detection accuracy of more than 90% for shallow adhesive defects.

[0140] Based on this, the defect segmentation model provided in this application includes an encoder and a decoder. The implementation process of step 230 can be as follows: input the image block to be inspected and the standard image block into the encoder, and obtain the first multi-scale feature map of the image block to be inspected at multiple preset resolutions and the second multi-scale feature map of the standard image block at multiple resolutions through the encoder; input the first multi-scale feature map and the second multi-scale feature map into the decoder, and perform feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder to output the defect segmentation image block.

[0141] It should be noted that each resolution corresponds to one first feature map and one second feature map, and multiple resolutions correspond to multiple first feature maps (i.e., first multi-scale feature maps) and multiple second feature maps (second multi-scale feature maps).

[0142] As an example, see Figure 4 The diagram shown illustrates the structure of the defect detection model. This diagram only illustrates the processing flow of one image block to be inspected and one standard image block. The specific processing flow after multiple image blocks to be inspected and multiple standard image blocks are input into the defect segmentation model is similar and will not be described in detail here.

[0143] The encoder consists of four Transformer blocks (Transformer Block 1, Transformer Block 2, Transformer Block 3, and Transformer Block 4). After passing through each Transformer block, the feature maps are downsampled to 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. These four feature maps at different resolutions are then fused in the decoder. The decoder can be implemented using a convolutional neural network, performing bilinear interpolation and 3x3 convolution operations on all feature maps in the encoder to achieve the effect of transposed convolution, while avoiding the checkerboard effect that can negatively impact the network model.

[0144] Specifically, in the decoder, feature maps of different sizes are first upsampled to the original image size, and then concatenated to obtain a concatenated feature map containing high-resolution coarse-grained features and low-resolution fine-grained features. Further, this concatenated feature map is input into a subsequent convolutional network, where three convolutional operations reduce the dimensionality of the feature map to 64, 32, and 16 dimensions, respectively. Finally, the 16-dimensional convolutional feature map is input into the final convolutional layer, reducing the dimensionality to 2 dimensions.

[0145] Here, 2D indicates whether the current pixel has surface defects. For example, whether there is a shallow adhesive defect.

[0146] Based on the above explanation, as Figure 5 As shown, the process of pre-training the defect segmentation model in this application may include the following steps:

[0147] Step 510: Obtain training sample images of the inspected products.

[0148] Among them, the inspected products are those whose surface defects have been marked. The inspected products and the products to be inspected mentioned above can be of the same type or different types.

[0149] In this step, the training sample images include a standard sample image and multiple defect sample images. Each sample image carries a sample label, which is used to indicate whether the sample image has surface defects.

[0150] It should be noted that for multiple defect sample images of the same type of product, multiple standard sample images can also be combined to form training sample pairs. This application embodiment does not limit the number of standard sample images.

[0151] As an example, for inspected product A, multiple defect sample images pre-detected for the inspected product can be acquired, along with two standard sample images of product A. The two standard sample images differ slightly in shooting angle, lighting information, and other aspects.

[0152] Step 520: Generate multiple training sample pairs based on the training sample images.

[0153] Each training sample pair consists of two sample images.

[0154] In one possible implementation, such as Figure 6 As shown, the implementation process of step 520 can be as follows:

[0155] Step 521: Combine the standard sample image with multiple defect sample images in sequence to obtain multiple initial sample pairs;

[0156] That is, each initial sample pair includes a defective sample image and a standard sample image.

[0157] Step 523: Randomly perform data augmentation on multiple initial sample pairs to obtain multiple training sample pairs.

[0158] In one possible implementation, the data augmentation process includes at least one of the following:

[0159] (1) Randomly swap the two sample images in the initial sample pair to obtain multiple first sample pairs;

[0160] It should be noted that each initial sample pair can be represented as (sample image 1, sample image 2), that is, for the defect segmentation model, there is a sequential order between the two sample images in the initial sample pair.

[0161] As an example, in order to model the difference between two input images from the training strategy, when inputting defective sample image D and standard sample image R, the initial sample pair (D, R) is randomly swapped, and the defect label is not changed, so that f(D, R) = f(R, D) holds, thus conforming to the modeling of the difference between two input images.

[0162] (2) According to the preset pixel offset range, perform pixel offset processing on any sample image in the initial sample pair to obtain multiple second sample pairs.

[0163] In the preprocessing of sample image registration, due to the stability of the algorithm used, hardware limitations and speed requirements, defective sample images and standard sample images may not achieve pixel-level alignment, resulting in an offset of more than 5 pixels.

[0164] Therefore, in order to enable the deep learning network model to adapt to this situation, when training the initial defect detection model, a random offset operation is performed on the defect sample image or standard sample image in the initial sample pair, so that there is a positional offset between the two sample images.

[0165] In this way, by simulating common image shift phenomena in real-world scenarios, the final trained defect segmentation model can be made more robust.

[0166] (3) Randomly copy the standard sample image in the initial sample pair to the defect sample image to obtain multiple third sample pairs.

[0167] In other words, the third sample pair includes two identical defect sample images. At this time, no matter how the images are offset or rotated, the defect label corresponding to the third sample pair should be a matrix of all zeros, indicating that there is no difference between the two sample images and therefore no defect.

[0168] (4) Paste the pre-extracted image defects into the standard sample image of the initial sample pair to obtain multiple fourth sample pairs.

[0169] Specifically, surface defects already labeled in the historical detection dataset are extracted and randomly pasted onto the standard sample image of the initial sample pair to obtain randomly generated defect sample images. Then, the original standard sample image in the initial sample pair and the randomly generated defect sample image are used to form a fourth sample pair to alleviate the data requirements of the deep learning model.

[0170] Based on this, the multiple training sample pairs used to train the initial defect segmentation model in the embodiments of this application include at least one of the following: the initial sample pair, the first sample pair, the second sample pair, the third sample pair, and the fourth sample pair.

[0171] Step 530: Input multiple training sample pairs into the initial defect segmentation model to be trained, and calculate the training loss value of the target loss function of the initial defect segmentation model based on the output of the initial defect segmentation model;

[0172] In one possible implementation, step 530 can be implemented as follows: Multiple training sample pairs are divided into a training set and a validation set. Training sample pairs from the training set are input into the initial defect segmentation model in batches, allowing the initial defect segmentation model to fully learn the feature information of both standard sample images and defect sample images. After training the training set for a preset number of iterations, training sample pairs from the validation set are used to test the learning effect of the initial defect segmentation model, and the training loss value of the target loss function of the initial defect segmentation model is calculated.

[0173] Both the training set and the validation set include multiple training sample pairs. The number of training sample pairs in the two sets may be the same or different, and this application embodiment does not impose any restrictions on this.

[0174] It should be understood that the training loss is determined by the difference between the pixel information in the defect segmentation image patch predicted by the initial defect segmentation model and the pre-labeled defect labels of the input training samples. When the difference decreases, it means that the training result is close to the labeled information, and the training can be terminated.

[0175] As an example, the preset number of iteration training rounds can be any number, such as 10 rounds, 20 rounds, etc.

[0176] Furthermore, for shallow adhesive defects, they only occupy a small portion of the entire single color box image. Therefore, when performing pixel-level classification, there may be a class imbalance problem. Although the data augmentation processing described above can alleviate some of the problem, the embodiments of this application still need to use an objective loss function to constrain the class imbalance, thereby avoiding missed detections.

[0177] In one possible implementation, the target loss function of the initial defect segmentation model includes the Dice loss function and the cross-entropy loss function, and the coefficients of the Dice loss function and the cross-entropy loss function are different in the target loss function.

[0178] It should be understood that Dice is a region-related loss function used to calculate the similarity between two samples. It performs well in scenarios with severe imbalance between positive and negative samples and focuses more on the discovery of foreground regions during training. Cross-entropy measures the degree of difference between two different probability distributions in the same random variable. In machine learning, it is represented as the difference between the true probability distribution and the model's predicted probability distribution. The smaller the cross-entropy value, the better the model's prediction effect.

[0179] As an example, the target loss function of the initial defect segmentation model is set using the Dice loss function and the CrossEntropy loss function, as shown in the following formula (1):

[0180] Loss=1.0*Dice(A,A′)+0.1*CrossEntropy(A,A′) (1)

[0181] The Dice loss function and the cross-entropy loss function are used to calculate the pixel position deviation, as shown in formulas (2) and (3) below.

[0182] Dice(A,A′)=1-2|A&A′| / (|A|+|A′|) (2)

[0183] CrossEntropy(A,A′)=-\sum A′log(A) (3)

[0184] In the formula, A′ represents the defect segmentation result output by the initial defect segmentation model, that is, the pixel position information of the shallow glued flower defect on the defect sample image; A represents the position information of the shallow glued flower defect that is manually pre-annotated in the defect sample image; the coefficient of the Dice loss function is 1, and the coefficient of the cross-entropy loss function is 0.1.

[0185] Thus, by calculating the corresponding pixel position deviations using the Dice loss function and the cross-entropy loss function respectively, and then superimposing the loss weights (i.e. calculating the coefficients) to set the target loss function of this application, the problem of severe imbalance between positive and negative samples can be solved.

[0186] Step 540: If the training loss value does not meet the preset convergence condition, adjust the parameters of the initial defect segmentation model, and input multiple training sample pairs into the adjusted initial defect segmentation model again until the training loss value meets the preset convergence condition. Then, end the training and obtain the trained defect segmentation model.

[0187] Adjusting the parameters of the initial defect segmentation model includes adjusting the processing parameters of each Transformer block in the initial defect segmentation model, as well as the network parameters of the convolutional neural network.

[0188] As an example, the preset convergence conditions include: the training loss value reaches a preset loss threshold; the training loss value no longer changes in the preset number of iterative training rounds; and the total number of training rounds reaches a preset total number of training rounds.

[0189] For example, the loss threshold can be 0.1, the number of iterations can be 10, and the total number of training iterations can be 100.

[0190] Optionally, before deploying the trained defect segmentation model, some product surface images can be acquired as a test set to test the accuracy of the defect segmentation image blocks predicted by the defect segmentation model. If the test results meet the requirements, the trained defect segmentation model can assist in surface defect detection in real-world scenarios.

[0191] In the above-described defect segmentation model training process, this embodiment of the application obtains multiple training sample pairs by randomly performing data augmentation processing on the initial sample pairs, thereby improving the diversity and increasing the number of training sample pairs, and making the final trained defect segmentation model more robust. Simultaneously, this embodiment of the application uses the Dice loss function and the cross-entropy loss function to set the target loss function of the initial defect segmentation model, improving the segmentation accuracy of the defect segmentation model and avoiding missed detections.

[0192] Step 240: Based on the defect segmentation image patch, determine whether the product under inspection has surface defects.

[0193] In this context, the defect segmentation image block represents the pixel difference information between an image block to be inspected and a standard image block. In other words, the pixels in the defect segmentation image can reflect whether surface defect information exists in the surface image to be inspected compared to the standard surface image. The surface defect information includes information such as the defect location, size, and contour.

[0194] It should be understood that the number of defect segmentation image blocks is the same as the number of image blocks to be inspected or standard image blocks input into the defect segmentation model. For a defect segmentation image block, the number of pixels included may be 0 or may not be 0.

[0195] If the number of pixels in the defect segmentation image block is 0, it means that there is no pixel difference between the image block to be inspected and the standard image block as determined by the defect segmentation model; if the number of pixels in the defect segmentation image block is not 0, it means that there is a difference between the image block to be inspected and the standard image block as determined by the defect segmentation model at these non-zero pixels.

[0196] In one possible implementation, step 240 can be implemented as follows: based on the defect segmentation image block, determine whether there are pixels describing defect information in each defect segmentation image block; if there are no pixels describing defect information in the defect segmentation image block, i.e., the defect segmentation image block is empty, then determine that there is no surface defect in the image region corresponding to the defect segmentation image block in the surface image to be inspected; if there are pixels describing defect information in the defect segmentation image block, i.e., the defect segmentation image block is not empty, then determine that there is a surface defect in the image region corresponding to the defect segmentation image block in the surface image to be inspected.

[0197] Specifically, if there are no pixels describing defect information in the defect segmentation image blocks corresponding to all the image blocks to be inspected in the surface image to be inspected, it is determined that there is no surface defect in the surface image to be inspected, and the corresponding product to be inspected also has no surface defect, and the product to be inspected is a good product.

[0198] Furthermore, if there are pixels describing defect information in the defect segmentation image block, the position information of these pixels in the defect segmentation image block can be determined, and the position information can be mapped onto the surface image to be inspected, thereby determining the specific location, size, outline and other defect information of the surface defect.

[0199] In another possible implementation, step 240 can be implemented as follows: obtain pixel intersection information between the defect segmentation image block and the candidate defect feature map; if the pixel intersection information is empty, it is determined that there is no surface defect on the surface of the product to be inspected; if the pixel intersection information is not empty, it is determined that there is a surface defect on the surface of the product to be inspected.

[0200] It should be understood that when two pixels have different pixel values, the difference in their pixel values ​​can reflect the difference between them. Therefore, when determining whether a product under inspection has surface defects based on defect segmentation image blocks, the candidate defect feature map obtained by subtracting pixel values ​​from pixels at the same position in the surface image under inspection and the standard surface image, as well as performing connected component analysis, can be combined to improve the accuracy of surface defect detection results.

[0201] Therefore, if there are pixels describing defect information in both the defect segmentation image block and the candidate defect feature map, then after taking the intersection, the pixel intersection information will include these pixels describing defect information.

[0202] Furthermore, if the pixel intersection information is not empty, the position information of each pixel in the intersection in the defect segmentation image block or candidate defect feature map can be determined, and the position information can be mapped to the surface image to be inspected, so as to determine the specific location, size, outline and other defect information of the surface defect.

[0203] In this embodiment, after acquiring the surface image of the product to be inspected and a standard surface image, the computer device first performs image segmentation on the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image. The number of image blocks to be inspected and standard image blocks are the same, and the image regions occupied by the image blocks to be inspected and the standard image blocks in their respective surface images are also the same. Then, the image blocks to be inspected and the standard image blocks are input into a pre-trained defect segmentation model. The defect segmentation model obtains a defect segmentation image block corresponding to one image block to be inspected and one standard image block. This defect segmentation image block reflects the pixel difference information between the image block to be inspected and the standard image block. Finally, based on the defect segmentation image block, it is determined whether the product to be inspected has surface defects. Therefore, this application utilizes a pre-trained defect segmentation model for defect segmentation, which is more efficient than manual inspection. Moreover, the defect segmentation model is a dual-input neural network model, with its inputs including a standard image patch corresponding to a standard surface image and a standard image patch corresponding to the surface to be inspected. This addresses the problem of weak generalization ability in deep learning models, eliminating the need for model training for different scenarios and products, thus reducing model training costs. The dual-input defect segmentation model can maintain good defect detection results even when encountering products not present in the training dataset. Furthermore, this application pre-segments the surface image to be inspected and the standard surface image, obtaining at least one image patch corresponding to the surface to be inspected and at least one standard image patch corresponding to the standard surface image. Then, the defect segmentation model processes each image patch, reducing resource consumption during image processing and lowering the computational performance requirements of the computer equipment when segmenting image patches. This allows the surface defect detection method to achieve high-speed real-time computation on low-end computing devices, improving the efficiency of surface defect detection.

[0204] Based on the above embodiments, in one exemplary embodiment, as follows: Figure 7 As shown, this application also provides another surface defect detection method, which can be applied to the above-mentioned... Figure 1 The computer device 100 shown is used as an example to illustrate the implementation process of this surface defect detection method as follows:

[0205] The process involves acquiring an initial image of the product to be inspected and an initial standard image. First, based on the initial standard image, image registration is performed on the initial image to be inspected to obtain an intermediate image. Then, the Sobel operator is used to perform edge detection and edge dilation on the intermediate image and the initial standard image to obtain an image of the surface to be inspected and a standard surface image. Finally, feature comparison and Blob analysis are performed on the image of the surface to be inspected and the standard surface image to obtain N candidate defect feature maps; where N≥1.

[0206] Furthermore, based on the regional location information of each candidate defect feature map, the candidate defect feature maps are sorted according to their image area size, and image segmentation is performed on the surface image to be inspected and the standard surface image based on each candidate defect feature map to obtain N image blocks to be inspected and N standard image blocks.

[0207] Among them, there are N image blocks to be inspected and N standard image blocks that correspond one-to-one. The corresponding image blocks to be inspected and standard image blocks describe the same surface area on the product to be inspected.

[0208] Furthermore, N image blocks to be inspected and N standard image blocks are input into the dual-input defect segmentation model provided in this application in the order of image block pairs to obtain defect segmentation image blocks corresponding to each pair of input image blocks to be inspected and standard image blocks.

[0209] Among them, the defect segmentation image block is used to describe the pixel difference information between an image block to be inspected and a standard image block. The pixels included may be empty, that is, there is no pixel difference between the image block to be inspected and the standard image block; the pixels included may also be non-empty, that is, there is a pixel difference between the image block to be inspected and the standard image block.

[0210] It should be understood that after N image blocks to be inspected and N standard image blocks are processed by the defect segmentation model, N defect segmentation image blocks are obtained.

[0211] Furthermore, the intersection of the N defect segmentation image blocks and the corresponding N candidate defect feature maps is taken to determine whether each defect segmentation image block has surface defects.

[0212] Finally, the surface defect judgment results of each defect segmentation image block are screened to determine whether the product under inspection has surface defects.

[0213] It should be noted that when implementing the above defect detection method in this application embodiment, the implementation principle and technical effect can be found in the relevant content of steps 210-240 in the previous embodiment, and will not be repeated here.

[0214] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0215] Based on the above-described surface defect detection method and employing the same technical concept, this application also provides a surface defect detection device for implementing the above-described surface defect detection method. The solution provided by this device is similar to the solution described in the above-described method embodiments. Therefore, the specific limitations in one or more surface defect detection device embodiments provided below can be found in the limitations of the surface defect detection method described above, and will not be repeated here.

[0216] In one exemplary embodiment, such as Figure 8 As shown, the surface defect detection device 800 includes:

[0217] Image acquisition module 810 is used to acquire images of the surface to be inspected and standard surface of the product to be inspected.

[0218] The image segmentation module 820 is used to segment the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image; the number of image blocks to be inspected and standard image blocks are the same, and the image regions of the image blocks to be inspected and standard image blocks in the corresponding surface images are also the same.

[0219] The defect segmentation module 830 is used to input the image block to be inspected and the standard image block into a pre-trained defect segmentation model to obtain at least one defect segmented image block; each defect segmented image block is used to represent the pixel difference information between an image block to be inspected and a standard image block.

[0220] The defect detection module 840 is used to determine whether the product under inspection has surface defects based on defect segmentation image blocks.

[0221] In one possible implementation, the image segmentation module 820 includes:

[0222] The pixel processing unit is used to subtract the pixel values ​​of pixels at the same position in the surface image to be inspected and the standard surface image to obtain an intermediate surface image.

[0223] The region analysis unit is used to perform connected region analysis on the intermediate surface image to obtain at least one candidate defect feature map.

[0224] The image segmentation unit is used to segment the surface image to be inspected and the standard surface image according to the region location information of the candidate defect feature map in the intermediate surface image, so as to obtain the image block to be inspected and the standard image block.

[0225] In one possible implementation, the defect detection module 840 includes:

[0226] The information acquisition unit is used to acquire pixel intersection information between defect segmentation image blocks and candidate defect feature maps;

[0227] The first determining unit is used to determine that there are no surface defects on the surface of the product to be inspected if the pixel intersection information is empty.

[0228] The second determining unit is used to determine that there is a surface defect on the surface of the product to be inspected if the pixel intersection information is not empty.

[0229] In one possible implementation, the defect segmentation model includes an encoder and a decoder; then the defect segmentation module 830 includes:

[0230] The feature extraction unit is used to input the image block to be inspected and the standard image block into the encoder, and to obtain the first multi-scale feature map of the image block to be inspected at multiple preset resolutions and the second multi-scale feature map of the standard image block at multiple resolutions through the encoder.

[0231] The feature fusion unit is used to input the first multi-scale feature map and the second multi-scale feature map into the decoder, and the decoder performs feature fusion processing on the first multi-scale feature map and the second multi-scale feature map to output defect segmentation image blocks.

[0232] In one possible implementation, the image acquisition module 810 includes:

[0233] The image acquisition unit is used to acquire the initial image of the product to be inspected and the initial standard image of the product to be inspected.

[0234] The image registration unit is used to perform image registration processing on the initial image to be inspected based on the initial standard image to obtain the intermediate image to be inspected.

[0235] The image processing unit is used to perform image preprocessing on the intermediate image to be inspected and the initial standard image respectively, so as to obtain the surface image to be inspected and the standard surface image.

[0236] In one possible implementation, the image processing unit is specifically used for:

[0237] Edge detection is performed on the intermediate image to be inspected and the initial standard image to obtain the first edge information of the intermediate image to be inspected and the second edge information of the initial standard image.

[0238] Based on the first edge information, edge dilation processing is performed on the middle image to be inspected to obtain the surface image to be inspected.

[0239] Based on the second edge information, the initial standard image is subjected to edge dilation processing to obtain a standard surface image.

[0240] In one possible implementation, such as Figure 9 As shown, the surface defect detection device 800 also includes:

[0241] The sample acquisition module 850 is used to acquire training sample images of the inspected products. The training sample images include a standard sample image and multiple defect sample images. Each sample image carries a sample label, which is used to indicate whether the sample image has surface defects.

[0242] The sample pair preparation module 860 is used to generate multiple training sample pairs based on the training sample images; each training sample pair includes two sample images.

[0243] The loss calculation module 870 is used to input multiple training sample pairs into the initial defect segmentation model to be trained, and calculate the training loss value of the target loss function of the initial defect segmentation model based on the output of the initial defect segmentation model.

[0244] The model training module 880 is used to adjust the parameters of the initial defect segmentation model if the training loss value does not meet the preset convergence condition, and then input multiple training sample pairs into the initial defect segmentation model after parameter adjustment again until the training loss value meets the preset convergence condition, then the training ends and a trained defect segmentation model is obtained.

[0245] In one possible implementation, the sample pair preparation module includes:

[0246] The sample combination unit is used to combine a standard sample image with multiple defective sample images in sequence to obtain multiple initial sample pairs;

[0247] The sample pair processing unit is used to randomly perform data augmentation on multiple initial sample pairs to obtain multiple training sample pairs.

[0248] Data augmentation processing includes at least one of the following:

[0249] The two sample images in the initial sample pair are randomly swapped to obtain multiple first sample pairs;

[0250] According to the preset pixel offset range, pixel offset processing is performed on any one of the sample images in the initial sample pair to obtain multiple second sample pairs;

[0251] The standard sample image in the initial sample pair is randomly copied as a defective sample image to obtain multiple third sample pairs;

[0252] The pre-extracted image defects are pasted into the standard sample images of the initial sample pairs to obtain multiple fourth sample pairs;

[0253] Multiple training sample pairs include at least one of the following: initial sample pair, first sample pair, second sample pair, third sample pair, and fourth sample pair.

[0254] In one possible implementation, the target loss function of the initial defect segmentation model includes the Dice loss function and the cross-entropy loss function, and the coefficients of the Dice loss function and the cross-entropy loss function are different in the target loss function.

[0255] In one possible implementation, if the product to be inspected is a printed product, then surface defects include shallow adhesive defects.

[0256] It should be noted that, Figure 8 and Figure 9 The various modules in the surface defect detection device 800 shown can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0257] Furthermore, it should be understood that the embodiments of this application can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes a computer program. When the computer program is loaded and run on a computer device, all or part of the processes or functions shown in the embodiments of this application are generated.

[0258] The computer program may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program may be transferred from one website, terminal, server or data center to another website, terminal, server or data center via wired or wireless means.

[0259] The computer-readable storage medium can be any available medium that can be accessed by a computer device, or a data storage device such as a server or data center that integrates one or more available media.

[0260] It should be understood that the above are merely specific implementation methods of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A surface defect detection method characterized by, The method comprises the following steps: acquiring a surface image of a product to be inspected and a standard surface image; performing image segmentation on the surface image of the product to be inspected and the standard surface image to acquire at least one image block of the product to be inspected and at least one image block of the standard surface image; the number of the image blocks of the product to be inspected and the standard image blocks is the same, and the image blocks of the product to be inspected and the standard image blocks are located in the same image area in the corresponding surface image; inputting the image blocks of the product to be inspected and the standard image blocks into a pre-trained defect segmentation model to acquire at least one defect segmentation image block; each defect segmentation image block is used to represent the pixel difference information between one image block of the product to be inspected and one standard image block; determining whether the surface of the product to be inspected has a defect based on the defect segmentation image block; the image segmentation on the surface image of the product to be inspected and the standard surface image to acquire at least one image block of the product to be inspected and at least one image block of the standard surface image comprises the following steps: performing a pixel value subtraction operation on the pixels at the same position in the surface image of the product to be inspected and the standard surface image to obtain an intermediate surface image; performing a connected region analysis on the intermediate surface image to acquire at least one candidate defect feature map; performing image segmentation on the surface image of the product to be inspected and the standard surface image according to the area position information of the candidate defect feature map in the intermediate surface image to obtain the image blocks of the product to be inspected and the standard image blocks; the determination of whether the surface of the product to be inspected has a defect based on the defect segmentation image block comprises the following steps: acquiring pixel intersection information between the defect segmentation image block and the candidate defect feature map; if the pixel intersection information is empty, it is determined that the surface of the product to be inspected does not have a defect; if the pixel intersection information is not empty, it is determined that the surface of the product to be inspected has a defect; the defect segmentation model comprises an encoder and a decoder; the inputting of the image blocks of the product to be inspected and the standard image blocks into the pre-trained defect segmentation model to acquire at least one defect segmentation image block comprises the following steps: inputting the image blocks of the product to be inspected and the standard image blocks into the encoder to acquire first multi-scale feature maps of the image blocks of the product to be inspected and second multi-scale feature maps of the standard image blocks at a plurality of preset resolutions through the encoder; inputting the first multi-scale feature maps and the second multi-scale feature maps into the decoder to perform feature fusion processing on the first multi-scale feature maps and the second multi-scale feature maps through the decoder to output the defect segmentation image block.

2. The method of claim 1, wherein, the acquisition of the surface image of the product to be inspected and the standard surface image comprises the following steps: acquiring an initial image of the product to be inspected and an initial standard image; performing image registration processing on the initial image of the product to be inspected based on the initial standard image to obtain an intermediate image of the product to be inspected; performing image preprocessing on the intermediate image of the product to be inspected and the initial standard image respectively to obtain the surface image of the product to be inspected and the standard surface image.

3. The method of claim 2, wherein, The image preprocessing is performed on the intermediate inspection image and the initial standard image respectively to obtain the inspection surface image and the standard surface image, including: Edge detection is performed on the intermediate inspection image and the initial standard image to obtain first edge information of the intermediate inspection image and second edge information of the initial standard image; Based on the first edge information, edge dilation processing is performed on the intermediate inspection image to obtain the inspection surface image; Based on the second edge information, edge dilation processing is performed on the initial standard image to obtain the standard surface image.

4. The method of claim 1, wherein, The training process of the defect segmentation model includes: obtaining training sample images of inspected products; the training sample images include one standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used to indicate whether the sample image has a surface defect; According to the training sample images, a plurality of training sample pairs are generated; each training sample pair includes two sample images; The plurality of training sample pairs are input into an initial defect segmentation model to be trained, and a training loss value of a target loss function of the initial defect segmentation model is calculated according to an output of the initial defect segmentation model; If the training loss value does not satisfy a preset convergence condition, the parameters of the initial defect segmentation model are adjusted, and the plurality of training sample pairs are input into the initial defect segmentation model with adjusted parameters again until the training loss value satisfies the preset convergence condition, and the training is ended to obtain the trained defect segmentation model.

5. The method of claim 4, wherein, The plurality of training sample pairs are generated according to the training sample images, including: The standard sample image is combined with the plurality of defect sample images in sequence to obtain a plurality of initial sample pairs; The plurality of initial sample pairs are randomly subjected to data enhancement processing to obtain the plurality of training sample pairs; The data enhancement processing includes at least one of the following: The two sample images in the initial sample pair are randomly exchanged to obtain a plurality of first sample pairs; According to a preset pixel offset range, pixel offset processing is performed on any sample image in the initial sample pair to obtain a plurality of second sample pairs; The standard sample image in the initial sample pair is randomly copied as a defect sample image to obtain a plurality of third sample pairs; An image defect extracted in advance is pasted into the standard sample image of the initial sample pair to obtain a plurality of fourth sample pairs; The plurality of training sample pairs include at least one of the initial sample pair, the first sample pair, the second sample pair, the third sample pair, and the fourth sample pair.

6. The method of claim 4, wherein, The target loss function of the initial defect segmentation model includes a Dice loss function and a cross-entropy loss function, and the calculation coefficients of the Dice loss function and the cross-entropy loss function in the target loss function are different.

7. The method of claim 1, wherein, If the inspected product is a printed matter, the surface defect includes a shallow sticky flower defect.

8. A surface defect detection apparatus characterized by comprising: It includes: An image acquisition module is configured to acquire an inspection surface image and a standard surface image of an inspected product; An image segmentation module is configured to perform image segmentation on the to-be-inspected surface image and the standard surface image to obtain at least one to-be-inspected image block corresponding to the to-be-inspected surface image and at least one standard image block corresponding to the standard surface image; the to-be-inspected image block and the standard image block have the same number and are located in the same image region in the corresponding surface image; A defect segmentation module is configured to input the to-be-inspected image block and the standard image block into a pre-trained defect segmentation model to obtain at least one defect segmentation image block; each defect segmentation image block is used to represent pixel difference information between the to-be-inspected image block and the standard image block; A defect detection module is configured to determine whether the to-be-inspected product has a surface defect based on the defect segmentation image block. The image segmentation on the to-be-inspected surface image and the standard surface image to obtain at least one to-be-inspected image block corresponding to the to-be-inspected surface image and at least one standard image block corresponding to the standard surface image comprises: performing pixel value subtraction on pixels at the same position in the to-be-inspected surface image and the standard surface image to obtain an intermediate surface image; performing connected region analysis on the intermediate surface image to obtain at least one candidate defect feature map; performing image segmentation on the to-be-inspected surface image and the standard surface image according to region position information of the candidate defect feature map in the intermediate surface image to obtain the to-be-inspected image block and the standard image block; The determination whether the to-be-inspected product has a surface defect based on the defect segmentation image block comprises: obtaining pixel intersection information between the defect segmentation image block and the candidate defect feature map; if the pixel intersection information is empty, determining that the to-be-inspected product surface does not have a surface defect; if the pixel intersection information is not empty, determining that the to-be-inspected product surface has a surface defect. The defect segmentation model comprises an encoder and a decoder. The inputting of the to-be-inspected image block and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmentation image block comprises: inputting the to-be-inspected image block and the standard image block into the encoder to obtain, through the encoder, a first multi-scale feature map of the to-be-inspected image block at a plurality of preset resolutions and a second multi-scale feature map of the standard image block at the plurality of resolutions; inputting the first multi-scale feature map and the second multi-scale feature map into the decoder to perform feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder and output the defect segmentation image block. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8. The processor invokes and executes the computer program from the memory to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images