Image defect detection method and related equipment

Abstract

The present application discloses an image defect detection method and related equipment. The method includes: segmenting the image to be detected to obtain multiple first image blocks; inputting the multiple first image blocks into a multi-target detection model to obtain key points of the image to be detected; the multi-target detection model is obtained by training multiple samples; mapping the key points to the image to be detected to obtain a target detection image; clustering multiple key points in the target detection image to obtain a defect area corresponding to at least one image defect; segmenting the defect area in the target detection image to obtain multiple third image blocks, the third image block includes the defect area; inputting the multiple third image blocks into the multi-target detection model to determine the image defects of the image to be detected. Through the above method, image defects can be quickly detected and the accuracy of detecting image defects can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image detection technology, and in particular, relates to image defect detection and related equipment. Background Art

[0002] Since the images of problematic products may have many defects, relying on manual inspection is not only inefficient but may also result in missed inspections. Therefore, a defect detection method based on machine vision is used to detect product defects.

[0003] At present, the defect detection method based on machine vision generally uses the detection model trained based on the sample set to detect the product image. However, the above defect detection method has the problem of poor accuracy. Summary of the invention

[0004] In view of this, an image defect detection method and related equipment provided in an embodiment of the present application can quickly detect image defects and improve the accuracy of detecting image defects.

[0005] In a first aspect, an embodiment of the present application provides an image defect detection method, the method comprising:

[0006] Segmenting the image to be detected to obtain a plurality of first image blocks;

[0007] Inputting the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected; the multi-target detection model is obtained by training based on a plurality of samples, wherein the samples include defective blocks containing defective areas and normal blocks not containing the defective areas, and the number of blocks of the defective blocks and the number of blocks of the normal blocks are in a preset ratio; the defective blocks and the normal blocks are obtained by segmenting the sample images;

[0008] Mapping the key points to the image to be detected to obtain a target detection image;

[0009] Clustering a plurality of key points in the target detection image to obtain a defect area corresponding to at least one image defect;

[0010] Segmenting the defective area in the target detection image to obtain a plurality of third image blocks, wherein the third image blocks include the defective area;

[0011] The plurality of third image blocks are input into the multi-target detection model to determine image defects of the image to be detected.

[0012] In some embodiments, inputting the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected includes:

[0013] Inputting the plurality of first image blocks into a multi-target detection model to obtain a heat map corresponding to a second image block, where the second image block is an image block containing image defects among the plurality of first image blocks;

[0014] The peak points in the thermal map are used as the key points of the image to be detected.

[0015] In some embodiments, segmenting the defect area in the target detection image to obtain a plurality of third image blocks includes:

[0016] Determining feature points of the defect area in the target detection image;

[0017] Based on the feature points of the defect area, the defect area in the target detection image is segmented according to a model segmentation size to obtain a plurality of third image blocks.

[0018] In some embodiments, before inputting the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected, the method further includes:

[0019] Obtain a training data set, where the training data set includes multiple sample images;

[0020] For each of the sample images, segment the sample image according to the model segmentation size to obtain the defective image block and the normal image block;

[0021] For each of the sample images, selecting the defective blocks and the normal blocks whose numbers are in the preset ratio as samples;

[0022] A multi-target detection model is trained based on the multiple samples.

[0023] In some embodiments, before segmenting the training data set according to the model segmentation size to obtain the defective image blocks and the normal image blocks, the method further includes:

[0024] Clustering the bounding boxes of the image defects in the training data set to obtain a plurality of bounding box groups;

[0025] Selecting a target bounding box group whose number of types of bounding boxes meets a preset condition from the multiple bounding box groups;

[0026] The size of the largest bounding box in the target bounding box group is used as the model segmentation size.

[0027] In some embodiments, for each of the sample images, segmenting the sample image according to the model segmentation size to obtain the defective image block and the normal image block includes:

[0028] For each of the sample images, hierarchical clustering is performed on the image defects in the sample image to obtain a plurality of defect groups, wherein the defect group includes at least one of the image defects;

[0029] Based on the feature points in the defect group, segment the defect area corresponding to each defect group in the sample image according to the model segmentation size to obtain a plurality of defect blocks including the defect area;

[0030] The other regions in the sample image except the defective region are segmented according to the model segmentation size to obtain a plurality of normal image blocks that do not include the defective region.

[0031] In a second aspect, an embodiment of the present application provides an image defect detection device, the device comprising:

[0032] A first image segmentation module, used for segmenting the image to be detected to obtain a plurality of first image blocks;

[0033] A first input module is used to input the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected; the multi-target detection model is obtained by training based on a plurality of samples, wherein the samples include defective blocks containing defective areas and normal blocks not containing the defective areas, and the number of blocks of the defective blocks and the number of blocks of the normal blocks are in a preset ratio; the defective blocks and the normal blocks are obtained by segmenting the sample images;

[0034] A mapping submodule, used for mapping the key points to the image to be detected to obtain a target detection image;

[0035] A first clustering submodule, configured to cluster a plurality of key points in the target detection image to obtain a defect area corresponding to at least one image defect;

[0036] A first segmentation submodule, configured to segment the defect region in the target detection image to obtain a plurality of third image blocks, wherein the third image blocks include the defect region;

[0037] The defect determination submodule is used to input the multiple third image blocks into the multi-target detection model to determine the image defects of the image to be detected.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0039] When the processor executes the computer program instructions, the image defect detection method as described in the first aspect is implemented.

[0040] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the image defect method as described in the first aspect is implemented.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the image defect detection method as described in the first aspect.

[0042] The embodiments of the present application provide an image defect detection, model training method, device, electronic device, computer storage medium and computer program product, which obtains the key points of the image to be detected by dividing the image to be detected into small-sized first images and inputting multiple first images into a trained multi-target detection model. In this way, after dividing the large-size image into small-size images, inputting them into the multi-target detection model is conducive to reducing the model's video memory occupancy and improving the model detection efficiency. At the same time, the samples for training the multi-target detection model include defective blocks and normal blocks with a preset number of blocks, so the problem of model inaccuracy caused by imbalance of positive and negative samples can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of the principle of an image defect training and testing method provided by the prior art;

[0045] Figure 2 A schematic diagram of another image defect training and testing method provided in the prior art;

[0046] Figure 3 It is a flowchart of an image defect detection method provided according to an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the principle of a multi-target detection model provided in an embodiment of the present application;

[0048] Figure 5 It is a flowchart of another image defect detection method provided by an embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of the principle of a hierarchical clustering method provided in an embodiment of the present application;

[0050] Figure 7A schematic diagram of the principle of a multi-target detection model training method provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the principle of a multi-target detection model training method provided in an embodiment of the present application;

[0052] Fig. 9 is a structural schematic diagram of an image defect detection device provided in an embodiment of the present application;

[0053] Fig.10 It is a structural schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0055] Before describing in detail an image defect detection method provided by an embodiment of the present application, a brief introduction to existing image defect detection technologies is first given.

[0056] The bounding box (bbox) is the outer rectangular box of the target defect in image defect detection, which is used to identify the target defect.

[0057] Currently, a detection model based on a neural network is used to detect defects in low-resolution images. However, when using ultra-high-resolution images with more than 50 million pixels to train the detection model, the following problems exist:

[0058] 1. When inputting ultra-high-resolution images into the detection model, the resolution of the ultra-high-resolution images is too large. Therefore, the video memory usage is too high during the operation of the detection model, and it may not be trained normally on a processor with only 16G video memory.

[0059] 2. Due to the large variation in defect scale, it is difficult for a detection model to take into account both large and small defects at the same time.

[0060] 3. It takes a long time to train the detection model.

[0061] In order to solve the above problems, the entire image is usually divided into multiple small images for training during the defect detection training of ultra-high resolution images. This avoids the problem of being unable to train normally due to the image being too large. On the other hand, all the details of the image are retained to ensure that tiny defects in ultra-high resolution images can be detected, so that the detection model has the ability to distinguish small defects.

[0062] Figure 1 The following is a schematic diagram of the principle of an image defect training and testing method provided by the prior art. Figure 1 As shown, a fixed-size sliding window is used to segment the ultra-high-resolution image into multiple fixed-size image blocks, wherein each image block has no overlap or each image block has overlap. Multiple image blocks are input into the defect detection model for training and testing of the defect detection model.

[0063] However, in the above scheme, on the one hand, since there are many segmented image blocks, the time for training the model and the time for defect detection are relatively long.

[0064] On the other hand, the defective area accounts for a very small proportion of the overall image, and a small number of image blocks are positive samples with defects, while most image blocks are negative samples without defects. Inputting all image blocks into the detection model for training will lead to an extreme imbalance of positive and negative samples when training the model, resulting in an increase in the number of missed detections by the detection model.

[0065] On the other hand, due to the lack of overall information of the image, the trained detection model has weaker detection capabilities for large defects exceeding a fixed size.

[0066] On the other hand, the regression box in the segmented image block may be obtained by cutting the regression box on the original image, so there may be incomplete defect information in the segmented image block, or there may be useless regression box information in the image block. In this way, using the above image blocks to train the detection model will lead to the problem of increased over-detection of the detection model.

[0067] Figure 2 The following is a schematic diagram of another image defect training and testing method provided by the prior art. Figure 2 As shown in the figure, the regression frame of each annotation frame in the ultra-high resolution image is traversed, and the regression frame is used as the center coordinate of the cut image to perform center cut on the original image, or the image is cut after an appropriate offset at the center point. The cut image block is input into the defect detection model to train the defect detection model. During the test, the ultra-high resolution image is segmented with an overlapping sliding window, and the segmented image block is input into the monitoring model for defect detection.

[0068] For example, for an image with a resolution of 10000x10000, the image defect area is segmented into image blocks with a resolution of 512*512 according to the center point of the regression box of a single defect. In this way, as many image blocks as there are image defects in the original image are segmented for model training.

[0069] On the one hand, the above solution solves the problem of high image resolution leading to inability to train normally by performing center cuts. On the other hand, all details of the image defects are retained, so that the detection model has the ability to distinguish tiny defects. On the other hand, only image blocks containing image defects are used to train the detection model, which can not only solve the problem of incomplete defect information or useless regression box information in the image blocks, but also reduce the impact of negative samples.

[0070] However, the above solution only uses positive samples for training, which may result in the trained detection model being unable to distinguish negative samples well, resulting in an increase in over-detection of the detection model.

[0071] Moreover, in the above scheme, segmentation is performed according to the regression box of each defect. When multiple defects exist at the same time and are close to each other, a part of another defect may be cut off, and there is no annotation information of this part in each image block, causing the detection model to recognize another defect as background information, which may lead to an increase in the detection model's missed detection of the defect.

[0072] Furthermore, due to the lack of overall image information in the above scheme, the detection model's ability to distinguish large defects becomes weak.

[0073] In addition, the detection results are inaccurate due to the different splitting methods of the training process and the testing process.

[0074] In view of this, the present application proposes an image defect detection method, device, electronic device, computer storage medium and computer program product, which can obtain a target detection model based on defective blocks and normal blocks proportional to the number of blocks, so that the multi-target detection model can not only learn the defect information of positive samples, but also take into account the defect-free information of negative samples, thereby enabling the multi-target detection model to accurately detect image defects.

[0075] The image defect detection method provided in the embodiment of the present application can be executed by an image defect detection device. Figure 3 FIG. 1 is a flow chart of an image defect detection method provided according to an embodiment of the present application. Figure 3 As shown, the image defect detection method includes:

[0076] Step S11, segmenting the image to be detected to obtain a plurality of first image blocks.

[0077] Step S12: input the multiple first image blocks into a multi-target detection model to obtain key points of the image to be detected.

[0078] Here, the multi-target detection model is obtained by training based on multiple samples, wherein the samples include defective blocks containing defective areas and normal blocks not containing defective areas, and the number of defective blocks and normal blocks is a preset ratio. The defective blocks and the normal blocks are obtained by segmenting the sample images.

[0079] Step S13, mapping the key points to the image to be detected to obtain a target detection image.

[0080] Step S14: clustering multiple key points in the target detection image to obtain a defect area corresponding to at least one image defect.

[0081] Step S15, segmenting the defect area in the target detection image to obtain a plurality of third image blocks.

[0082] Step S16: input the plurality of third image blocks into a multi-target detection model to determine image defects of the image to be detected.

[0083] In the above embodiment, by dividing the image to be detected into small-sized first image blocks, and inputting multiple first images into the trained multi-target detection model, the key points of the image to be detected are obtained. In this way, after the large-size image is divided into small-size images, the input into the multi-target detection model is conducive to reducing the model's video memory occupancy and improving the model detection efficiency. At the same time, the samples for training the multi-target detection model include defective blocks and normal blocks with a preset number of blocks, so the problem of model inaccuracy caused by imbalance of positive and negative samples can be avoided.

[0084] The specific implementation of the above steps will be described in detail below.

[0085] In step S11, the image defect detection device can segment the image to be detected according to a sliding window of a fixed size, thereby obtaining a plurality of first image blocks. The image to be detected can be an ultra-high resolution image or an image of other resolutions. In the embodiment of the present application, there is no restriction on the type of the image to be detected.

[0086] Optionally, the fixed size may be a model segmentation size. The model segmentation size may be a size of samples used to train a multi-target detection model.

[0087] The "first image block" involved in the embodiment of the present application may be of a fixed size. The first image block may be a partial image of the image to be detected. And each first image block does not contain overlapping parts.

[0088] The "defective block" involved in the embodiments of the present application may be an image block containing a defective area, wherein the defective area is determined by clustering image defects in the sample image. The "normal block" is an image block that does not contain a defective area and may be obtained by segmenting a non-overlapping sliding window.

[0089] The "multi-target detection model" involved in this application can use an end-to-end detection model to determine the target key points and regress certain attributes of the target based on the positions of the key points, such as the target size, regression box size, and target category. The multi-target detection model includes but is not limited to CornerNet, ExtremeNet, and CenterNet models.

[0090] In the embodiment of the present application, the structure of the multi-target detection model is illustrated by taking CenterNet as an example. Figure 4 is a schematic diagram of the principle of a multi-target detection model provided in an embodiment of the present application, such as Figure 4 As shown, the multi-target detection model includes two parts: key point extraction and coordinate regression. Among them, the backbone network of the target detection model, namely the keypoint backbone, can be a standard network such as Resnet18, DLA-34, Hourglass-104, etc., which is used to extract image key points. Regression mainly includes: center point bias regression, and regression box size regression. The output network of the multi-target detection model is mainly composed of three branch networks, and the three branch networks respectively output the center point, the bias value of the center point coordinates and the predicted size of the regression box. The entire model will output C+4 values ​​at each position, namely the key point category C, the offset x, y, the size w, h, and all outputs share a fully convolutional backbone. Among them, the center point represents the target, and the center point output network generates a heat map, which is used to identify classification information. Each category has a heat map. As shown Figure 4 In the figure, 128*128*80 represents the heat map corresponding to 80 categories.

[0091] In some embodiments, the loss function of the multi-target detection model may be a cross entropy loss function FocalLoss.

[0092] It should be noted that the loss function of the multi-target detection model in the embodiment of the present application is not limited to FocalLoss.

[0093] The objective loss function can be:

[0094] L det =L k +λ size L size +λ off Loff (1)

[0095] Among them, λ size is the weight corresponding to the size loss function, λ off is the weight corresponding to the bias loss function, L det Represents the total loss function, L size is the size loss function.

[0096] L k Represents the loss function of the key point, L k It can be expressed as:

[0097]

[0098] Among them, Y xyc is the true value, indicating the probability of whether the pixel is a key point. When the pixel is a key point, Y xyc is 1. is the predicted value, α and β are the hyperparameters of focal loss, and N is the number of key points in an image.

[0099] L off As the center point bias loss function, the regression loss function L1loss can be used. Its expression can be:

[0100]

[0101] in, represents the predicted value of the center point offset, N is the number of key points in an image, R is the size scaling ratio, Indicates the position of the center point on the feature map, Represents the predicted value of the center point location.

[0102] It should be noted that when the image is downsampled, the key points will have deviations due to the discrete data, so a center point bias loss function is set.

[0103] L size is the regression box size loss function, and the regression loss function L1loss can also be used. Its expression can be expressed as:

[0104]

[0105] in, is the estimated regression box size, S k is the actual regression box size, and N is the number of key points.

[0106] The "multi-target detection model" in the embodiment of the present application can be trained by multiple samples. Specifically, it can be trained by other devices and then provided to the image defect detection device, or it can be trained by the image defect detection device itself.

[0107] Figure 5 It is a flowchart of another image defect detection method provided in an embodiment of the present application.

[0108] like Figure 5 As shown, before step S12, the image defect detection method further includes:

[0109] Step S21, obtaining a training data set, where the training data set includes a plurality of sample images.

[0110] Here, the sample image includes the annotation information of the image defects. The training data set can be obtained by collecting multiple sample images as the training data set, such as obtaining the original data set from a database or a network, and then preprocessing the images in the training data set to obtain images that meet the input standard of the multi-target detection model.

[0111] Among them, preprocessing may include graying, geometric transformation, image enhancement, denoising, interpolation, rotation, rotation and flipping, etc. Preprocessing can be set according to the multi-target detection model, or preprocessing rules can be pre-set and stored in the image defect detection device, so that the image defect detection device can process the image according to the preset preprocessing rules, thereby improving the reliability of extracting sample features.

[0112] Step S22 : for each sample image, segment the sample image according to the model segmentation size to obtain defective blocks and normal blocks.

[0113] Here, the image defect detection device can use a sliding window to segment each sample image, thereby obtaining defective blocks and normal blocks. The size of the sliding window is the model segmentation size. The defective block is a block containing a defective area, and the normal block is a normal block containing a defective area. The defective area is the area corresponding to the image defect in the sample image.

[0114] Step S23 : for each of the sample images, select defective blocks and normal blocks whose numbers are in the preset ratio as samples.

[0115] Here, for each sample image, all defective blocks and some normal blocks are selected as samples for training the multi-target detection model, wherein the number of defective blocks and normal blocks is in a preset ratio. Optionally, the preset ratio may be 1:1.

[0116] It should be noted that the preset ratio in the embodiment of the present application may include but is not limited to 1:1, or may be other ratios, such as 1:3.

[0117] In some embodiments, the normal image block used for training the model may be a normal image block near the defective image block. In this way, the information around the image defect is used for training the model, which is helpful to improve the accuracy of the model.

[0118] Step S24, training a multi-target detection model based on multiple samples.

[0119] Here, multiple samples are input into the multi-target detection model for model training. In this way, the model training of defect location and defect classification is completed using multiple samples.

[0120] In an embodiment of the present application, defective blocks and normal blocks with a preset ratio of blocks are used as samples, and the samples are used to train the model. In this way, the model can learn the defect information of the positive samples and take into account the background information of the negative samples. Therefore, on the one hand, the problem of imbalance between positive and negative samples due to the small proportion of positive samples in the training samples, which increases the missed detection of the trained model, can be avoided. On the other hand, the problem of using only positive samples for training models, which results in the model being unable to distinguish negative samples well and increases the over-detection of the trained model, can be avoided, thereby making the positive and negative samples used for training the model balanced.

[0121] In some embodiments, in order to retain most defects after the image is segmented into small image blocks and avoid selecting a suitable segmentation size through repeated experiments, before step S22, the method further includes:

[0122] Clustering the bounding boxes of image defects in the training data set to obtain multiple bounding box groups;

[0123] Selecting a target bounding box group whose number of bounding box types meets a preset condition from the multiple bounding box groups;

[0124] The size of the largest bounding box in the target bounding box group is used as the model segmentation size.

[0125] Here, clustering the bounding boxes of image defects in the training data set may be performed using any clustering method. Optionally, clustering the regression boxes of image defects in the training data set includes performing hierarchical clustering on the bounding boxes of image defects in the training data set.

[0126] Figure 6 Schematic diagram of the principle of a hierarchical clustering method provided in the embodiment of the present application. Figure 6As shown in Figure 1, in hierarchical clustering, the distance between an object in cluster C1 and an object in cluster C2 is the smallest of all the Euclidean distances between objects belonging to different clusters, and C1 and C2 may be merged. Hierarchical clustering is a single-link method, each cluster can be represented by all the objects in the cluster, and the similarity between two clusters is determined by the similarity of the closest pair of data points in the two clusters.

[0127] In an embodiment of the present application, the sample images in the training set include annotation information of image defects, and the annotation information includes information of a bounding box. The information of the bounding box may include size and position information of the bounding box. The bounding box grouping includes bounding boxes of multiple defect categories. The preset condition may be the three groups with the largest number of bounding box types, or the top n groups. The target bounding box grouping may be the three groups with the largest number of bounding box types, or the top n groups of bounding box groupings.

[0128] That is to say, the embodiment of the present application may select the size of the largest bounding box in the first n groups with the largest number of bounding box types as the model segmentation size.

[0129] In the embodiment of the present application, the model segmentation size is used to segment the sample image when training the model, and to segment the image to be detected when detecting image defects.

[0130] In the present embodiment, after clustering the bounding boxes of image defects, target bounding box groups that meet preset conditions are screened, and the largest bounding box is selected from the target bounding box groups, and the largest size of the largest bounding box is used as the model segmentation size. In this way, a suitable segmentation size can be obtained adaptively, which solves the problem of repeated debugging of the segmentation size during training, and most defects can be retained in the image block during segmentation.

[0131] In some embodiments, in order to accurately segment defective blocks and normal blocks, step S22 may be specifically performed as follows:

[0132] For each sample image, hierarchical clustering is performed on the image defects in the sample image to obtain multiple defect groups;

[0133] Based on the feature points in the defect group, segment the defect area corresponding to each defect group in the sample image according to the model segmentation size to obtain a plurality of defect blocks containing the defect area;

[0134] The other regions in the sample image except the defective region are segmented according to the model segmentation size to obtain a plurality of normal image blocks that do not contain the defective region.

[0135] Here, the defect group includes at least one of the image defects. Hierarchical clustering of the image defects of the sample image may be hierarchical clustering of the center points of the image defects marked in the sample image according to a specified distance to generate multiple defect groups. The feature points of each defect group are calculated, and the image defect area corresponding to the defect group is segmented according to the model segmentation size, so as to obtain multiple defect images containing defect areas.

[0136] In addition, in order to keep the sample size consistent, other regions in the sample image except the defective region may be segmented according to the model segmentation size, thereby obtaining a plurality of normal image blocks.

[0137] In the above embodiment, by clustering and grouping the sample images, the defect information on the defective blocks can be complete, avoiding the defective blocks after segmentation containing useless annotation information. In addition, image defects that are close to each other can be grouped together for segmentation, which can avoid the problem that when multiple image defects are close to each other, the multi-target detection model identifies some defects as background information because a segmented image block does not have annotation information for some adjacent image defects, so that the multi-target detection model misses more defects of this type.

[0138] In step S12, the "key point" can represent the location of the image defect in the image to be detected. The key point can be obtained through the backbone network of the multi-target detection model. There can be multiple key points in the image to be detected.

[0139] In some embodiments, in order to facilitate the determination of key points in the image to be detected, step S12 may be specifically performed as follows:

[0140] Inputting the plurality of first image blocks into a multi-target detection model to obtain a heat map corresponding to the second image block;

[0141] The peak points in the heat map are used as the key points of the image to be detected.

[0142] Here, the heat map can be a feature map, which refers to a visualization method in neural networks that helps understand which part of an image allows the convolutional neural network to make the final classification decision. The peak point of the heat map can be a probability peak point, that is, the probability peak point of the pixel belonging to an image defect.

[0143] The "second image block" involved in the embodiment of the present application is an image block containing image defects among multiple first image blocks. Here, the second image block can be one or more. It should be noted that the second image block may include a complete image defect or an incomplete image defect.

[0144] In the embodiment of the present application, the backbone network of the multi-target detection model can be used to extract key points. Specifically, all first image blocks are input into the multi-target detection model to generate a heat map of the second image block, and the peak point of each defect category on each heat map is determined, and the peak point of each defect category is used as the key point.

[0145] In the above embodiment, by inputting multiple first image blocks into the multi-target detection model to extract the key points of the image to be detected, the problem of too high image resolution resulting in too much model memory usage can be solved, and since the size of the first image block is small, the time for extracting key points can be shortened.

[0146] In some embodiments, in order to detect complete image defect information in the image to be detected, step S13 may be specifically performed as follows:

[0147] In step S13, the target detection image may be a fusion image of the image to be detected and the thermal image containing the key points. Mapping the key points to the image to be detected may be performed by superimposing the thermal image containing the key points on the image to be detected.

[0148] In step S14, the "clustering" involved in the embodiment of the present application can be implemented by using any clustering method. Optionally, clustering multiple key points in the target detection image can include performing hierarchical clustering on the multiple key points in the target detection image.

[0149] In the embodiment of the present application, multiple key points in the target detection image are clustered, and key points with close distances can be grouped into the same cluster group. In this way, key points belonging to the same image defect can be grouped into the same group, and image defects with close distances can be grouped into the same group. In this way, the defect area corresponding to each cluster group is obtained, wherein each defect area can be an image area corresponding to at least one image defect.

[0150] In step S15, the "third image block" involved in the embodiment of the present application may include a defect area. The image defect detection device segments the defect area obtained after key point clustering from the target detection image, thereby obtaining a plurality of third image blocks.

[0151] In step S16, determining the image defect of the image to be detected may include determining the category of the image defect, the location of the image defect, and the size of the regression box. Inputting the third image block containing the defect area into the multi-target detection model can quickly detect the image defect in the image to be detected.

[0152] In the above embodiment, by mapping the key points to the original image, the position information of the key points in the original image can be obtained, and the key points are clustered and grouped, so that all the key points corresponding to each image defect can be obtained, which is beneficial to the model's recognition of large defects, and avoids partial segmentation of adjacent image defects, resulting in the model's inability to recognize adjacent image defects. At the same time, inputting the third image block containing the defect area into the multi-target detection model can avoid the multi-target detection model from detecting normal image blocks and reduce the number of defective image blocks detected by the multi-target detection model.

[0153] In order to improve the accuracy of model detection, the image segmentation size can be kept consistent when training the model and detecting image defects. In some embodiments, segmenting the defect area in the target detection image to obtain a plurality of third image blocks includes:

[0154] Determining feature points of a defect area in the target detection image;

[0155] Based on the feature points of the defective area, the defective area in the target detection image is segmented according to the model segmentation size to obtain a plurality of third image blocks.

[0156] Here, the feature point of the defect area may be the center point of the cluster group corresponding to the defect area. Taking the feature point of the defect area as the center, the defect area is segmented from the target detection image according to the model segmentation size to obtain multiple third image blocks of the same size.

[0157] In this way, the size of the image block input to the multi-target detection model for detection is consistent with the sample size for training the multi-target detection model, thereby improving the accuracy of image defect detection.

[0158] In order to further understand the training method of the multi-target detection model, the multi-target detection model is taken as the CenterNet model, and the clustering adopts the hierarchical clustering algorithm as an example to illustrate. Figure 7 The schematic diagram of the principle of a multi-target detection model training method provided in the embodiment of the present application. Figure 7 As shown, first, the image defect area in the sample image is clustered and grouped according to the automatically selected segmentation size.

[0159] Then, in the image data set containing image defects after clustering and grouping segmentation, normal image blocks that do not contain defects after scale segmentation are added in a certain proportion and sent to the multi-target detection network (Centernet) for defect location and defect classification model training.

[0160] Here, clustering and grouping segmentation refers to hierarchically clustering the center points of the marked defect boxes according to the specified distance and dividing them into multiple or one groups, then calculating the center point of each group and segmenting the defects according to the specified size.

[0161] Scale segmentation refers to cropping the length and width of the image in a sliding window according to a certain ratio, and the marked defect box is retained in the corresponding position of the cropped small image.

[0162] In the above embodiment, a multi-target detection model is trained using positive and negative samples in a certain proportion, so that the model can learn the defect information of the positive samples while taking into account the background information of the negative samples. It can also solve the problem of increased model missed detections due to the imbalance of positive and negative samples, as well as the problem of increased model overdetection due to training with only positive samples, thereby improving the accuracy of model training.

[0163] In addition, since the positive samples are obtained through clustering and grouping, they do not contain useless annotation information, thereby improving the integrity of image defect information.

[0164] In some embodiments, before clustering and grouping the sample images, it is necessary to automatically select the training set segmentation size (i.e., the model segmentation size). The segmentation size selection method is to hierarchically cluster the regression box of the training set (i.e., the bounding box of the image defect), and then find the largest regression box size in the three groups with the largest number of categories after clustering as the segmentation size. In this way, it can not only ensure that most defects are completely segmented, but also avoid the need for cumbersome operation steps to select the segmentation size.

[0165] In order to further understand the method of detecting image defects using a multi-target detection model, the multi-target detection model is taken as the CenterNet model, and the clustering is performed using a hierarchical clustering algorithm as an example. Figure 8 The schematic diagram of the principle of a multi-target detection model training method provided in the embodiment of the present application. Figure 8 As shown in the figure, first, the image to be detected is segmented into small images by sliding windows without overlap. Then all the small images are sent to the multi-target detection network to obtain the heat map, and the peak points in each small image are found through the heat map, and then the peak points are mapped back to the original image, and the peak points are clustered and grouped on the original image, and then sent to the multi-target detection network (Centernet) for defect location and defect classification.

[0166] In the above embodiment, the peak points in the segmented small image are extracted, and the peak points are mapped back to the original image, so as to obtain the complete image defect information in the original image, which is conducive to the model's recognition of large defects and can improve the accuracy of image defect positioning. And the peak points are clustered and grouped to obtain the small image to be detected, so that the small image can retain the complete image defects. The small image is input into the multi-target detection model for detection, which is conducive to reducing the video memory usage of the multi-target detection model, avoiding the monitoring of normal blocks, reducing the number of blocks to be detected, and improving the efficiency of model detection.

[0167] In practical applications, abrasions (Xs) and scratches (Hs) of an ultra-large resolution data set of 12000*8000 are detected, and the model effects and detection efficiency of the image defect detection method provided in the embodiment of the present application are compared with the two image defect detection methods provided in the prior art.

[0168] Detection results on the same test set:

[0169] Table 1 shows the test results of model 1

[0170] Label H Xs TotalNoOther precision 92.4％ 65.3％ 87.5％ recall 94.7％ 94.1％ 94.6％ ap 96.6％ 82.5％ 78.8％ f1 0.94 0.77 0.91 confidence 0.09 0.09 0.09 hit 2477 384 2861 pick 2681 588 3269 total 2617 408 3025 precent 86.5％ 13.5％ 100％

[0171] Table 2 shows the test results of Model 2

[0172] Label H Xs TotalNoOther precision 93.2％ 67.8％ 88.7％ recall 95.9％ 95.3％ 95.9％ ap 97.2％ 85.1％ 80.5％ f1 0.95 0.79 0.92 confidence 0.09 0.09 0.09 hit 2511 389 2900 pick 2695 574 3269 total 2617 408 3025 precent 86.5％ 13.5％ 100％

[0173] Table 3 Test results of model 3

[0174]

[0175]

[0176] Among them, Label represents label, precision represents accuracy, recall represents recall rate, ap represents average accuracy, f1 represents F1 value, confidence represents confidence, hit represents the number of correct detections, pick represents the number of detections, total represents the total number, precent represents the proportion of the number, and TotalNoOther represents the total.

[0177] It should be noted that model 1 is a sample trained by performing scale segmentation on the sample using a sliding window with coverage. Model 2 is a model trained by performing center segmentation of the bbox on the sample. Model 3 is a multi-target detection model provided in an embodiment of the present application.

[0178] It can be seen from Tables 1-3 that the detection results of Model 3 are better than those of Model 1 and Model 2. In addition, the average inference time per picture of Model 1 is 1.45 seconds, the average inference time per picture of Model 2 is 1.45 seconds, and the average inference time per picture of Model 3 is 1.23 seconds. It can be seen that the multi-target detection model provided in the embodiment of the present application is superior to other image defect detection methods in terms of model effect and detection efficiency under the same conditions.

[0179] Fig. 9 Schematic diagram of the structure of an image defect detection device provided in an embodiment of the present application. Fig. 9 As shown, the image defect detection device 30 includes:

[0180] A first image segmentation module 31 is used to segment the image to be detected to obtain a plurality of first image blocks;

[0181] A first input module 32 is used to input the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected; the multi-target detection model is obtained by training based on a plurality of samples, wherein the samples include defective blocks containing defective areas and normal blocks not containing the defective areas, and the number of blocks of the defective blocks and the number of blocks of the normal blocks are in a preset ratio; the defective blocks and the normal blocks are obtained by segmenting the sample images;

[0182] A mapping submodule 33 is used to map the key points to the image to be detected to obtain a target detection image;

[0183] A first clustering submodule 34 is used to cluster a plurality of key points in the target detection image to obtain a defect area corresponding to at least one image defect;

[0184] A first segmentation submodule 35 is used to segment the defect area in the target detection image to obtain a plurality of third image blocks, wherein the third image blocks include the defect area;

[0185] The defect determination submodule 36 is used to input the plurality of third image blocks into the multi-target detection model to determine image defects of the image to be detected.

[0186] In some embodiments, the first input module 32 includes:

[0187] a heat map generation submodule, configured to input the plurality of first image blocks into a multi-target detection model to obtain a heat map corresponding to a second image block, where the second image block is an image block containing image defects in the plurality of first image blocks;

[0188] The first submodule is used to use the peak point in the thermal map as the key point of the image to be detected.

[0189] In some embodiments, the first segmentation submodule is specifically used to determine the feature points of the defect area in the target detection image; based on the feature points of the defect area, the defect area in the target detection image is segmented according to the model segmentation size to obtain multiple third image blocks.

[0190] In some embodiments, before the first input module 32, the image defect detection device further includes:

[0191] An acquisition module is used to acquire a training data set, where the training data set includes a plurality of sample images;

[0192] A first segmentation module is used to segment each sample image according to the model segmentation size to obtain the defective image block and the normal image block;

[0193] A selection module, configured to select, for each of the sample images, the defective blocks and the normal blocks whose numbers are in the preset ratio as samples;

[0194] A training module is used to train a multi-target detection model based on a plurality of the samples.

[0195] In some embodiments, before the first segmentation module, the image defect detection device 30 further includes:

[0196] A first clustering module, configured to cluster the bounding boxes of image defects in the training data set to obtain a plurality of bounding box groups;

[0197] A selection submodule, configured to select a target bounding box group whose number of types of bounding boxes meets a preset condition from the plurality of bounding box groups;

[0198] The second submodule is used to use the size of the largest bounding box in the target bounding box group as the model segmentation size.

[0199] In some embodiments, the first segmentation module is specifically used to perform hierarchical clustering on the image defects in the sample image for each of the sample images to obtain multiple defect groups, where the defect group includes at least one of the image defects; based on the feature points in the defect group, the defect area corresponding to each defect group in the sample image is segmented according to the model segmentation size to obtain multiple defect blocks containing the defect area; and the other areas in the sample image except the defect area are segmented according to the model segmentation size to obtain multiple normal blocks that do not contain the defect area.

[0200] Fig.10 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application, see Fig.10 , the electronic device comprises:

[0201] The electronic device may include a processor 41 and a memory 42 storing computer program instructions.

[0202] Specifically, the processor 41 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0203] The memory 42 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 42 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 42 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 42 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 42 is a non-volatile solid-state memory.

[0204] The memory 42 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present application.

[0205] The processor 41 implements any one of the image defect detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 42 .

[0206] In one example, the electronic device may further include a communication interface 43 and a bus 44. Fig.10 As shown, the processor 41, the memory 42, and the communication interface 43 are connected via a bus 44 and communicate with each other.

[0207] The communication interface 43 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0208] Bus 44 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 44 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0209] The electronic device can implement the method in the embodiment of the present application to achieve the combination Figures 3 to 9 An image defect detection method and apparatus are described.

[0210] In addition, in combination with the image defect detection method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image defect detection methods in the above embodiment is implemented.

[0211] In addition, in combination with the image defect detection method in the above-mentioned embodiment, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implements any one of the image defect detection methods in the above-mentioned embodiments when the computer program or instructions are executed by a processor.

[0212] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0213] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0214] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0215] The above reference is to the flowchart and / or block diagram of the method, vehicle-mounted terminal, electronic device and computer program product according to the embodiment of the present application, and describes various aspects of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0216] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image defect detection method, characterized in that: The method comprises: Segmenting the image to be detected to obtain a plurality of first image blocks; Inputting the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected; the multi-target detection model is obtained by training based on a plurality of samples, wherein the samples include defective blocks containing defective areas and normal blocks not containing the defective areas, and the number of blocks of the defective blocks and the number of blocks of the normal blocks are in a preset ratio; the defective blocks and the normal blocks are obtained by segmenting the sample images; Mapping the key points to the image to be detected to obtain a target detection image; Clustering a plurality of key points in the target detection image to obtain a defect area corresponding to at least one image defect; Segmenting the defective area in the target detection image to obtain a plurality of third image blocks, wherein the third image blocks include the defective area; The plurality of third image blocks are input into the multi-target detection model to determine image defects of the image to be detected.

2. The method according to claim 1, characterized in that: The step of inputting the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected includes: Inputting the plurality of first image blocks into a multi-target detection model to obtain a heat map corresponding to a second image block, where the second image block is an image block containing image defects among the plurality of first image blocks; The peak points in the thermal map are used as the key points of the image to be detected.

3. The method according to claim 1, characterized in that The step of segmenting the defective area in the target detection image to obtain a plurality of third image blocks includes: Determining feature points of the defect area in the target detection image; Based on the feature points of the defect area, the defect area in the target detection image is segmented according to a model segmentation size to obtain a plurality of third image blocks.

4. The method according to claim 1, characterized in that: Before inputting the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected, the method further includes: Obtain a training data set, where the training data set includes multiple sample images; For each of the sample images, segment the sample image according to the model segmentation size to obtain the defective image block and the normal image block; For each of the sample images, selecting the defective blocks and the normal blocks whose numbers are in the preset ratio as samples; A multi-target detection model is trained based on the multiple samples.

5. The method according to claim 4, characterized in that Before segmenting the training data set according to the model segmentation size to obtain the defective image blocks and the normal image blocks, the method further includes: Clustering the bounding boxes of the image defects in the training data set to obtain a plurality of bounding box groups; Selecting a target bounding box group whose number of types of bounding boxes meets a preset condition from the multiple bounding box groups; The size of the largest bounding box in the target bounding box group is used as the model segmentation size.

6. The method according to claim 4, characterized in that For each of the sample images, segmenting the sample image according to the model segmentation size to obtain the defective image block and the normal image block includes: For each of the sample images, hierarchical clustering is performed on the image defects in the sample image to obtain a plurality of defect groups, wherein the defect group includes at least one of the image defects; Based on the feature points in the defect group, segment the defect area corresponding to each defect group in the sample image according to the model segmentation size to obtain a plurality of defect blocks including the defect area; The other regions in the sample image except the defective region are segmented according to the model segmentation size to obtain a plurality of normal image blocks that do not include the defective region.

7. An image defect detection device, characterized in that: The device comprises: A first image segmentation module, used for segmenting the image to be detected to obtain a plurality of first image blocks; A first input module is used to input the plurality of first image blocks into a multi-target detection model to obtain key points of the image to be detected; the multi-target detection model is obtained by training based on a plurality of samples, wherein the samples include defective blocks containing defective areas and normal blocks not containing the defective areas, and the number of blocks of the defective blocks and the number of blocks of the normal blocks are in a preset ratio; the defective blocks and the normal blocks are obtained by segmenting the sample images; A mapping submodule, used for mapping the key points to the image to be detected to obtain a target detection image; A first clustering submodule, configured to cluster a plurality of key points in the target detection image to obtain a defect area corresponding to at least one image defect; A first segmentation submodule, configured to segment the defect region in the target detection image to obtain a plurality of third image blocks, wherein the third image blocks include the defect region; The defect determination submodule is used to input the multiple third image blocks into the multi-target detection model to determine the image defects of the image to be detected.

8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image defect detection method according to any one of claims 1 to 6 is implemented.

9. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the image defect detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the image defect detection method according to any one of claims 1 to 6 is implemented.

Images