A defect detection method, device, electronic device and storage medium

Abstract

The present application provides a defect detection method, apparatus, electronic device and storage medium, which are used to improve the problem of low accuracy in detecting defects on item packages. The method includes: obtaining a to-be-detected image, which is obtained by photographing a target item package; using a defect detection model to perform defect detection on the to-be-detected image to obtain a defect detection result, where the defect detection model is trained using a sample image, a sample label and an active contour map corresponding to the sample image, the sample label represents that the category of the sample image is a defective image or a non-defective image, the active contour map corresponding to the defective image is an image marked with a package contour and a defect center position, and the active contour map corresponding to the non-defective image is an image marked with a package contour.

Description

Technical Field

[0001] This application relates to the technical fields of image processing and deep learning. Specifically, it relates to a defect detection method, device, electronic device, and storage medium. Background Art

[0002] Currently, the main method for detecting defects on the packaging of items (such as biscuit foods and small toy items, etc.) is an algorithm that manually extracts features from a large number of packaging images. The main process of this algorithm includes: first, selecting a region of interest from the packaging image, then extracting features from the region that may contain an object in the region of interest, and finally classifying the extracted features to determine whether there are defects on the packaging of the item. However, in the specific practice process, it is found that the current accuracy rate for detecting defects on the packaging of items is relatively low. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a defect detection method, device, electronic device, and storage medium, which are used to improve the problem of relatively low accuracy rate for detecting defects on the packaging of items.

[0004] The embodiments of this application provide a defect detection method, including: obtaining an image to be detected, where the image to be detected is obtained by photographing the packaging of a target item; using a defect detection model to perform defect detection on the image to be detected to obtain a defect detection result. The defect detection model is trained using a sample image, a sample label, and an active contour map corresponding to the sample image. The sample label represents whether the category of the sample image is a defective image or a non-defective image. The active contour map corresponding to the defective image is an image marked with the packaging contour and the defect center position, and the active contour map corresponding to the non-defective image is an image marked with the packaging contour. In the implementation process of the above solution, the defect detection model obtained by training with the active contour map performs defect detection on the image to be detected. Since the active contour map can guide the defect detection model to pay more attention to the packaging contour and the defect center position during the training stage, the accuracy rate of the trained defect detection model is greatly improved. Therefore, the defect detection model obtained by training with the active contour map can improve the accuracy rate of detecting defects on food packaging.

[0005] Optionally, in the embodiments of this application, the defect detection model includes: an encoder and a classifier; using the defect detection model to perform defect detection on the image to be detected to obtain a defect detection result, including: using the encoder to perform feature extraction on the image to be detected to obtain a feature mapping matrix; using the classifier to classify the feature mapping matrix to obtain a defect detection result.

[0006] In the implementation process of the above solution, since the trained defect detection model pays more attention to the packaging contour and the defect center position, therefore, the encoder trained with the active contour map is used to extract features from the image to be detected, and the trained classifier is used to classify the feature mapping matrix, so as to improve the accuracy of classifying and detecting defects on food packaging.

[0007] Optionally, in the embodiment of the present application, the defect detection model further includes: a decoder; after obtaining the feature mapping matrix, it further includes: using the decoder to predict the active contour map corresponding to the image to be detected according to the feature mapping matrix.

[0008] In the implementation process of the above solution, since the trained defect detection model pays more attention to the packaging contour and the defect center position, therefore, the encoder trained with the active contour map is used to extract features from the image to be detected, and the decoder trained with the active contour map is used to predict the active contour map corresponding to the image to be detected according to the feature mapping matrix, so as to improve the accuracy of classifying and detecting defects on food packaging.

[0009] Optionally, in the embodiment of the present application, before using the defect detection model to detect defects in the packaging image to be detected, it further includes: obtaining a sample image, a sample label, and the active contour map corresponding to the sample image; using the sample image and the active contour map corresponding to the sample image to train the encoder and the decoder, and using the sample image and the sample label to train the encoder and the classifier, and obtaining the defect detection model through the way of joint training.

[0010] In the implementation process of the above solution, the encoder and the decoder in the defect detection model are trained through the active contour map, so that the trained defect detection model pays more attention to the packaging contour and the defect center position, thereby improving the accuracy of classifying and detecting defects on food packaging.

[0011] Optionally, in the embodiment of the present application, obtaining a sample image, a sample label, and the active contour map corresponding to the sample image includes: obtaining the sample image and the sample label corresponding to the sample image; if the sample label corresponding to the sample image is a defective image, then mark the packaging contour and the defect center position in the sample image to obtain a first marked image, and generate the active contour map corresponding to the sample image according to the first marked image; if the sample label corresponding to the sample image is a non-defective image, then mark the packaging contour in the sample image to obtain a second marked image, and generate the active contour map corresponding to the sample image according to the second marked image.

[0012] In the implementation process of the above solution, by annotating the packaging contour and the position of the defect center in the sample image, a first annotated image is obtained, and an active contour map corresponding to the sample image is generated according to the first annotated image. Then, the active contour map is used to train the encoder and decoder in the defect detection model, so that the trained defect detection model pays more attention to the packaging contour and the position of the defect center, thereby improving the accuracy of classifying and detecting defects on food packaging.

[0013] Optionally, in the embodiment of the present application, training the encoder and decoder using the sample image and the active contour map corresponding to the sample image includes: using the encoder to extract features from the sample image to obtain a feature mapping matrix corresponding to the sample image; using the decoder to perform contour prediction on the feature mapping matrix corresponding to the sample image to obtain the predicted active contour map; calculating the active contour loss between the predicted active contour map and the active contour map corresponding to the sample image, and training the encoder and decoder according to the active contour loss.

[0014] In the implementation process of the above solution, by calculating the active contour loss between the predicted active contour map and the active contour map corresponding to the sample image, and training the encoder and decoder according to the active contour loss, the trained defect detection model pays more attention to the packaging contour and the position of the defect center, thereby improving the accuracy of classifying and detecting defects on food packaging.

[0015] Optionally, in the embodiment of the present application, training the encoder and classifier using the sample image and the sample label includes: using the encoder to extract features from the sample image to obtain a feature mapping matrix corresponding to the sample image; using the classifier to perform class prediction on the feature mapping matrix corresponding to the sample image to obtain the predicted class of the sample image; calculating the image classification loss between the predicted class of the sample image and the class of the sample label, and training the encoder and classifier according to the image classification loss.

[0016] The embodiment of the present application also provides a defect detection device, including: a detection image acquisition module, configured to acquire a to-be-detected image, where the to-be-detected image is obtained by photographing the packaging of a target item; a detection result acquisition module, configured to use the defect detection model to perform defect detection on the to-be-detected image to obtain a defect detection result, where the defect detection model is trained using a sample image, a sample label, and an active contour map corresponding to the sample image, the sample label represents that the category of the sample image is a defective image or a non-defective image, the active contour map corresponding to the defective image is an image annotated with the packaging contour and the position of the defect center, and the active contour map corresponding to the non-defective image is an image annotated with the packaging contour.

[0017] Optionally, in the embodiments of the present application, the defect detection model includes: an encoder and a classifier; the detection result obtaining module includes: a mapping matrix obtaining module, configured to use the encoder to perform feature extraction on the image to be detected to obtain a feature mapping matrix; a mapping matrix classification module, configured to use the classifier to classify the feature mapping matrix to obtain a defect detection result.

[0018] Optionally, in the embodiments of the present application, the defect detection model further includes: a decoder; the defect detection device further includes: an active contour prediction module, configured to use the decoder to predict an active contour map corresponding to the image to be detected according to the feature mapping matrix.

[0019] Optionally, in the embodiments of the present application, the defect detection device further includes: a sample data acquisition module, configured to acquire a sample image, a sample label, and an active contour map corresponding to the sample image; a detection model acquisition module, configured to use the sample image and the active contour map corresponding to the sample image to train the encoder and the decoder, and use the sample image and the sample label to train the encoder and the classifier, and obtain a defect detection model through a joint training method.

[0020] Optionally, in the embodiments of the present application, the sample data acquisition module includes: an image label acquisition module, configured to acquire a sample image and a sample label corresponding to the sample image; a first image annotation module, configured to, if the sample label corresponding to the sample image is a defective image, annotate the packaging contour and the defective center position in the sample image to obtain a first annotated image, and generate an active contour map corresponding to the sample image according to the first annotated image; a second image annotation module, configured to, if the sample label corresponding to the sample image is a non-defective image, annotate the packaging contour in the sample image to obtain a second annotated image, and generate an active contour map corresponding to the sample image according to the second annotated image.

[0021] Optionally, in the embodiments of the present application, the detection model acquisition module includes: an image feature extraction module, configured to use the encoder to perform feature extraction on the sample image to obtain a feature mapping matrix corresponding to the sample image; an image contour prediction module, configured to use the decoder to perform contour prediction on the feature mapping matrix corresponding to the sample image to obtain a predicted active contour map; a first model training module, configured to calculate an active contour loss between the predicted active contour map and the active contour map corresponding to the sample image, and train the encoder and the decoder according to the active contour loss.

[0022] Optionally, in the embodiments of the present application, the detection model acquisition module further includes: an image feature extraction module, configured to use an encoder to extract features from a sample image to obtain a feature mapping matrix corresponding to the sample image; an image category prediction module, configured to use a classifier to perform category prediction on the feature mapping matrix corresponding to the sample image to obtain a predicted category of the sample image; and a second model training module, configured to calculate an image classification loss between the predicted category of the sample image and the category of the sample label, and train the encoder and the classifier.

[0023] The embodiments of the present application further provide an electronic device, including: a processor and a memory, where the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the methods described above are performed.

[0024] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the methods described above are performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 A schematic flowchart of the defect detection method provided by the embodiments of the present application;

[0027] Figure 2 A schematic diagram of the active contour diagram corresponding to the food packaging provided by the embodiments of the present application;

[0028] Figure 3 A schematic diagram of the network structure of the encoder-decoder architecture provided by the embodiments of the present application;

[0029] Figure 4 A schematic diagram of a specific network structure of the defect detection model provided by the embodiments of the present application;

[0030] Figure 5 A schematic diagram of the training process of the defect detection model provided by the embodiments of the present application;

[0031] Figure 6 A schematic diagram of the structure of the defect detection device provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed embodiments of the present application, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the embodiments of the present application.

[0033] It can be understood that "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences.

[0034] Before introducing the defect detection method provided in the embodiments of the present application, some concepts involved in the embodiments of the present application will be introduced first:

[0035] A server refers to a device that provides computing services through a network. Servers include, for example, x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0036] It should be noted that the defect detection method provided in the embodiments of the present application can be executed by an electronic device. Here, the electronic device refers to a device terminal with the function of executing a computer program or the above-mentioned server. Device terminals include, for example, smart phones, personal computers, tablet computers, personal digital assistants, or mobile Internet devices, etc.

[0037] Next, the application scenarios applicable to this defect detection method will be introduced. Here, the application scenarios include but are not limited to: using this defect detection method to detect whether there are defects in the packaging of target items. Here, the target items include but are not limited to: food packaging (such as biscuits and potato chips), combined toy packaging (such as small building block toys and board games), solid chemical product packaging (such as small bags of fertilizers and pesticides), and industrial product packaging (other small parts such as screws). Since the defect detection model in this defect detection method is trained using data including active contour maps, etc., and the active contour map can guide the defect detection model to pay more attention to the packaging contour and the defect center position during the training stage, thereby improving the accuracy of detecting defects on food packaging.

[0038] The embodiments of the present application are described by taking the packaging of target items such as biscuits or potato chips as an example. When biscuits or potato chips are heat-sealed, they are likely to break into crumbs and bounce to the heat-sealing area. Moreover, the crumbs are small and concentrated in the heat-sealing area with relatively complex textures, resulting in that the slight abnormality of the packaging bag contour is more difficult to distinguish than the rupture or scratch in the middle area of the packaging bag. Therefore, it is almost impossible to synthesize samples by methods such as image enhancement to expand the defect dataset. Therefore, when defect detection is usually performed on food packaging such as biscuits or potato chips, there is likely a situation of insufficient quantity or imbalance of training samples (including sample images and sample labels). At this time, the defect detection method provided by the embodiments of the present application can be used to perform defect detection on food packaging such as biscuits or potato chips. Since the defect detection model in this defect detection method is trained using data including active contour maps, etc., and the active contour map can guide the defect detection model to pay more attention to the packaging contour and the defect center position during the training phase (i.e., providing a spatial attention mechanism for the defect detection model), the accuracy of detecting defects on food packaging is thus improved.

[0039] Please refer to Figure 1 the schematic flowchart of the defect detection method provided by the embodiments of the present application shown in; The embodiments of the present application provide a defect detection method, including:

[0040] Step S110: Obtain a to-be-detected image, which is obtained by photographing the packaging of a target item.

[0041] The to-be-detected image refers to an image that needs to be detected for defects. This image can be obtained by photographing the packaging of a target item. Here, the packaging of the target item can be any packaging with a relatively small volume and prone to subtle defects at the packaging seal. The target items include: food packaging, combined toy packaging, solid chemical product packaging, industrial product packaging, and so on.

[0042] Step S120: Use the defect detection model to perform defect detection on the to-be-detected image to obtain a defect detection result. The defect detection model is trained using a sample image, a sample label, and an active contour map corresponding to the sample image. The sample label represents whether the category of the sample image is a defective image or a non-defective image. The active contour map corresponding to the defective image is an image marked with the packaging contour and the defect center position, and the active contour map corresponding to the non-defective image is an image marked with the packaging contour.

[0043] Please refer to Figure 2Schematic diagram of the active contour diagram corresponding to the food packaging provided by the embodiment of the present application shown; the active contour diagram is an image that uses a contour representation method to represent the packaging contour and the center position of defects of the target item, that is, actively informing the neural network model of the contour information and defect position information of the target item in the input image. Specifically, for example, when representing the contour information and defect position information through gray values, each peak in the image contour represents a defect center (for example, here it is the protrusion where food debris bounces to the plastic seal). The specific calculation process of the active contour diagram will be introduced in detail below.

[0044] The defect detection model refers to a neural network model for defect detection of the image to be detected. This defect detection model can specifically adopt any neural network model with an encoder-decoder architecture, including but not limited to: the U-Net model or the multi-task defect detection model based on active contour. Hereinafter, the multi-task defect detection model based on active contour will be taken as an example for detailed description. Since this defect detection model is trained using data such as the active contour diagram, and the active contour diagram can provide a spatial attention mechanism for the defect detection model to improve the accuracy of defect detection.

[0045] In the implementation process of the above solution, the defect detection model obtained by training with the active contour diagram is used to detect defects in the image to be detected. Since the active contour diagram can guide the defect detection model to pay more attention to the packaging contour and the center position of defects during the training stage, the accuracy of the trained defect detection model is greatly improved. Therefore, the defect detection model obtained by training with the active contour diagram can improve the accuracy of detecting defects on food packaging.

[0046] Please refer to Figure 3 Schematic diagram of the network structure of the encoder-decoder architecture provided by the embodiment of the present application shown; the above multi-task defect detection model based on active contour can specifically adopt a multi-task model with an encoder-decoder architecture. This defect detection model can specifically include: an encoder, a decoder, and a classifier. Among them, the encoder is used to extract features from the input sample image to obtain the feature mapping matrix corresponding to the sample image (also known as the feature mapping diagram); the decoder is used to perform contour prediction on the feature mapping matrix to obtain the predicted active contour diagram. The decoder is retained during the model training stage, but during the model inference (also known as model inference) stage, the decoder can be retained or trimmed to reduce the computational amount of model inference; the classifier is used to perform category prediction on the feature mapping matrix to obtain the predicted category of the sample image.

[0047] As can be seen from the network structure diagram in the figure, the defect detection model mainly includes two tasks: a main task and an auxiliary task. The main task is to predict the category (defective or non-defective) of the target item packaging in the sample image through an encoder and a classifier. This task belongs to a binary classification task (i.e., Yes represents defective and No represents non-defective). The auxiliary task is to perform semantic segmentation and contour prediction on the sample image through an encoder and a decoder, so as to restore the active contour map corresponding to the sample image.

[0048] Please refer to Figure 4 A specific network structure diagram of the defect detection model provided by the embodiment of the present application shown in the figure; the size of the input image of the defect detection model can be adjusted according to the situation. Specifically, for example, the input image of the defect detection model is set to a color image with a size of 1024×512, then the image to be detected can be expressed as 1024×512×3, where 3 represents a three-channel color image. As can be seen from the above, the defect detection model can include: an encoder, a decoder, and a classifier; the encoder and the decoder can transfer feature maps in a splicing (Concate) manner, that is, the encoder and the decoder can achieve feature fusion through the splicing (Concate) method. The specific structures of the encoder, decoder, and classifier are introduced below.

[0049] The above-mentioned encoder can have multiple convolutional layers (Convolution Layer, abbreviated as Conv). The number of convolutional layers can be adjusted according to the specific situation. Specifically, for example, it includes a total of 7 convolutional layers, and the size of each convolutional layer is set to 4×4, and the corresponding convolutional kernels of the convolutional layers are 16, 32, 64, 128, 128, 128, and 128 in sequence. That is to say, the number of convolutional kernels corresponding to the convolutional layers in the encoder increases from 16 to 128, and its downsampling scale is 4. Among them, a batch normalization (Batch Normalization, BN) layer and an activation function (ActivationFunction) can be set behind each convolutional layer. Here, the activation function can adopt a leaky rectified linear unit (Leaky Rectified Linear Unit, Leaky_ReLU), and of course, other activation functions can also be set according to the specific situation.

[0050] The above decoder may have multiple deconvolution layers (abbreviated as DeConv or ConvT). Specifically, for example, it includes a total of 8 deconvolution layers, each with a size of 4×4, and the corresponding convolution kernels of the deconvolution layers are 128, 128, 128, 128, 64, 32, 16, and 1 in sequence. That is to say, the number of convolution kernels corresponding to the deconvolution layers in the decoder decreases from 128 to 16, and then from 16 to 1, and the upsampling scale is 4. Among them, a batch normalization (BN) layer and an activation function can be set behind most of the deconvolution layers (except the last one). The activation function here can use the rectified linear unit (ReLU), and of course, other activation functions can also be set according to specific situations. It can be understood that the last deconvolution layer is used to output the active contour map, and the size of the output active contour map can be set to 1024×512. A batch normalization (BN) layer and a hyperbolic tangent (tanh) activation function can also be set behind the last deconvolution layer.

[0051] The above classifier may include a convolution layer (abbreviated as Conv) and a fully connected layer (FC). The size of this convolution layer is set to 4×4, and the convolution kernel corresponding to the deconvolution layer is 128. This fully connected layer has 1024 hidden layer neurons.

[0052] As an alternative implementation of step S120, since the decoder can be retained or pruned during the model inference (also known as model inference) stage to reduce the computational complexity of model inference, the above defect detection model in the model inference stage may only include: an encoder and a classifier; the process of using the defect detection model to detect defects in the image to be detected may include:

[0053] Step S121: Use the encoder to extract features from the image to be detected to obtain a feature map matrix.

[0054] For example, the implementation of the above step S121: Assume that the image to be detected is represented as X, then the obtained feature map matrix can be expressed by the formula where E represents the feature extraction operation of the encoder, X represents the image to be detected, represents the feature map matrix.

[0055] Step S122: Use the classifier to classify the feature map matrix to obtain the defect detection result.

[0056] For example, in the implementation of the above step S122, assume that the classification operation of the classifier is represented by F. Then, the formula can be used to classify the feature mapping matrix to obtain the defect detection result of the classifier output (ClassificationOutput); where represents the feature mapping matrix, flatten represents the operation of reshaping the feature mapping matrix into a vector, F represents the classification operation of the classifier, represents the defect detection result output by the classifier.

[0057] As an alternative implementation of step S120, since the decoder can be retained or trimmed during the model inference (also known as model prediction) phase to reduce the computational complexity of model inference, the defect detection model in the above model inference phase may further include: a decoder; after obtaining the feature mapping matrix, it further includes:

[0058] Step S123: Use the decoder to predict the active contour map corresponding to the image to be detected according to the feature mapping matrix.

[0059] For example, in the implementation of the above step S123, assume that the active contour map prediction operation of the decoder is represented by D EDG , then the formula can be used to predict the active contour map corresponding to the image to be detected according to the feature mapping matrix; where represents the feature mapping matrix, D EDG represents the active contour map prediction operation of the decoder, represents the active contour map corresponding to the image to be detected predicted.

[0060] Please refer to Figure 5 the schematic diagram of the training process of the defect detection model provided in the embodiments of the present application shown; as an alternative implementation, before or after using the defect detection model, the defect detection model can also be trained. The training process of the defect detection model may include:

[0061] Step S210: Obtain a sample image, a sample label, and the active contour map corresponding to the sample image.

[0062] It can be understood that the above sample image may include: a defective image or a non-defective image. Therefore, the active contour map corresponding to the sample image can be generated by a program of a contour representation method according to the sample image, or can be manually labeled for the sample image.

[0063] Step S220: Train the encoder and decoder using the sample image and the corresponding active contour map of the sample image, and train the encoder and classifier using the sample image and the sample label. Obtain the defect detection model through joint training.

[0064] As an alternative implementation of step S210, the specific generation process of the above active contour map may include:

[0065] Step S211: Obtain the sample image and the corresponding sample label of the sample image.

[0066] The obtaining method of the above step S211 includes: The first obtaining method, after installing an industrial camera on the conveyor belt of the factory, the industrial camera installed on the conveyor belt can be used to take pictures of food packages (such as biscuit packages) on the conveyor belt. Each collected sample image contains only one packaging bag, and the sample image is obtained; then, the electronic device can store the sample image in the file system, database or mobile storage device. The second obtaining method, obtain the sample image from the file system, database or mobile storage device. The third obtaining method, obtain the sample image by using a browser or other application to access the Internet. The above sample label can be obtained by manual annotation. Specifically, for example: use 0 to mark the sample image without defects (i.e., the defect-free image), and use 1 to mark the sample image with defects (i.e., the defective image).

[0067] Step S212: If the sample label corresponding to the sample image is a defect-free image, annotate the packaging contour in the sample image to obtain a second annotated image, and generate the active contour map corresponding to the sample image according to the second annotated image.

[0068] The implementation manner of the above step S212 is, for example: If the sample label corresponding to the sample image is a defect-free image, the active contour method is used to set the packaging contour (i.e., the segmentation region contour between the background pixels and the target object) in the sample image and all pixel values inside the contour to a preset pixel value (for example, set to 127), and the remaining pixel values are set to the background pixel value (for example, set to 0 or 255) to obtain the second annotated image, and generate the active contour map corresponding to the sample image according to the second annotated image.

[0069] Step S212: If the sample label corresponding to the sample image is a defective image, annotate the packaging contour and the defect center position in the sample image to obtain a first annotated image, and generate the active contour map corresponding to the sample image according to the first annotated image.

[0070] The implementation manner of the above step S212 is, for example: If the sample label corresponding to the sample image is a defective image, the defective image can be expressed as Then, the active contour method is adopted to set the packaging contour in the sample image (i.e., the contour of the segmentation region between the background pixels and the target object) and the values of all pixel points within the contour to the initial pixel values (for example, set to 127). The initial pixel values here can be expressed as Then, assume that any point on the packaging contour (i.e., the contour of the segmentation region between the background pixels and the target object) is represented as N(x n ,y n ), and the center point of the defect is represented as O(x0, y0). Then, the points N(x n ,y n ) and O(x0, y0) can form a line segment ON. Then, any point (i.e., any point within the contour) on the line segment ON can be represented as X(x, y). According to the principle of linear interpolation, the ordinate value of the point X(x, y) can be obtained, and its ordinate value can be expressed as where x0 and y0 are the abscissa value and ordinate value of point O respectively, and x and y are the abscissa value and ordinate value of point X respectively. The values of the remaining background pixel points in the sample image are set to the background pixel values (for example, set to 0 or 255), and the center position of the defect is marked to obtain the first marked image.

[0071] Finally, use the formula to calculate the final pixel values of all pixel points within the contour; where represents the initial pixel value of the pixel points within the contour, h0 is the distance between any point N(x n ,y n ) on the packaging contour and the center point of the defect O(x0, y0), h1 is the distance between any point X(x, y) taken on the line segment ON and the point N(x n ,y n ), dilate represents the dilation operation in morphology, and the symbol "∧" represents the convolution kernel parameter in the dilation operation.

[0072] It can be understood that the dilation operation here is an optional operation. Without the dilation operation, the accuracy of defect detection on food packaging can also be improved. However, after the dilation operation, since the dilation operation can more vividly indicate and guide the defect detection model to pay more attention to the packaging contour and the center position of the defect, the accuracy of defect detection on food packaging can be further improved.

[0073] The technical principle by which the above defect detection model can improve the defect detection accuracy is that, in the above formula, the distance from any point within the contour region to the center point of the defect is used as the weight of the pixel value of that point, so that the defect detection model can easily perceive whether the pixel value of that point is at the peak or trough (i.e., the maximum or minimum value) of the active contour map. Therefore, when the defect detection model can accurately predict the active contour map, the model can easily learn the threshold for determining the peak, and thus locate the defect center position according to the threshold of the peak.

[0074] It can be understood that the above encoder and decoder, encoder and classifier can be trained separately in turns, or the encoder, decoder and classifier can be trained simultaneously.

[0075] In the first training method, the above encoder and decoder, encoder and classifier can be trained separately in turns. This training method may include:

[0076] As an optional implementation manner of step S220, the specific process of training the encoder and decoder may include:

[0077] Step S221: Use the encoder to extract features from the sample image to obtain the feature mapping matrix corresponding to the sample image.

[0078] Step S222: Use the decoder to perform contour prediction on the feature mapping matrix corresponding to the sample image to obtain the predicted active contour map.

[0079] Step S223: Calculate the active contour loss between the predicted active contour map and the active contour map corresponding to the sample image, and train the encoder and decoder according to the active contour loss.

[0080] The implementation manner of the above step S223 is, for example: use the formula to calculate the active contour loss between the predicted active contour map and the active contour map corresponding to the sample image; where X LSM represents the active contour map corresponding to the sample image, represents the predicted active contour map, |·| is the L1 norm, and l s represents the active contour loss (i.e., the loss of active contour segmentation). Then, update the network weight parameters of the encoder and the network weight parameters of the decoder in the defect detection model according to the active contour loss until the accuracy rate of the defect detection model no longer increases or the number of iterations (epoch) is greater than the preset threshold, and then the trained defect detection model can be obtained. Among them, the above preset threshold can also be set according to specific situations, such as set to 100 or 1000, etc.

[0081] As an alternative implementation of step S220, the specific process of training the encoder and the classifier described above may include:

[0082] Step S224: Use the encoder to extract features from the sample image to obtain a feature map matrix corresponding to the sample image.

[0083] Step S225: Use the classifier to perform class prediction on the feature map matrix corresponding to the sample image to obtain the predicted class of the sample image.

[0084] Step S226: Calculate the image classification loss between the predicted class of the sample image and the class of the sample label, and train the encoder and decoder according to the image classification loss.

[0085] The implementation manners of the above steps S224 to S226 are specifically as follows: The function formula can be used to calculate the image classification loss between the predicted class of the sample image and the class of the sample label; where l c represents the image classification loss, y i represents the class of the sample label corresponding to the i-th sample image, represents the predicted class of the i-th sample image. Then, train the encoder and decoder according to the image classification loss (that is, update the network weight parameters of the encoder and the network weight parameters of the decoder) until the accuracy rate of the defect detection model no longer increases or the number of iteration times (epoch) is greater than the preset threshold, and then the trained defect detection model can be obtained. Among them, the above preset threshold can also be set according to specific situations, such as setting it to 100 or 1000, etc.

[0086] The second training method is to train the encoder, decoder, and classifier simultaneously. This training method is similar to the first training method above. The difference is that the second training method needs to calculate the total loss value, which specifically may include:

[0087] Step S227: Calculate the total loss value according to the active contour loss and the image classification loss, and update the network weight parameters of the encoder, the network weight parameters of the decoder, and the network weight parameters of the classifier according to the total loss value.

[0088] The implementation manner of the above step S227 is, for example: Use the formula l = l c + w s · l s to calculate the active contour loss and the image classification loss to obtain the total loss value; where l represents the total loss value, l c represents the specific value of the image classification loss, w s represents the task weight value of the active contour loss, l sRepresents the specific value of the active contour loss. Then, update the network weight parameters of the encoder, the network weight parameters of the decoder, and the network weight parameters of the classifier according to the total loss value until the accuracy rate of the defect detection model no longer increases or the number of iterations is greater than the preset threshold, and then the trained defect detection model can be obtained. Among them, the above-mentioned preset threshold can also be set according to specific circumstances, such as set to 100 or 1000, etc.

[0089] Optionally, during the training process of the above-mentioned defect detection model, the Adam optimizer can also be used to optimize the training process of the defect detection model. During the process of using the Adam optimizer to optimize the model training, since the Adam optimizer absorbs the advantages of the gradient descent algorithm with adaptive learning rate and also absorbs the advantages of the momentum gradient descent algorithm, the Adam optimizer can not only adapt to sparse gradients but also alleviate the problem of gradient oscillation. Therefore, using the Adam optimizer to optimize the training process of the defect detection model can update the network weight parameters of the encoder, classifier, and decoder in the defect detection model together.

[0090] Please refer to Figure 6 The structural schematic diagram of the defect detection device provided by the embodiment of the present application shown. The embodiment of the present application provides a defect detection device 300, including:

[0091] A detection image acquisition module 310, configured to acquire a to-be-detected image, where the to-be-detected image is obtained by photographing the packaging of a target item.

[0092] A detection result acquisition module 320, configured to use the defect detection model to perform defect detection on the to-be-detected image to obtain a defect detection result. The defect detection model is trained using a sample image, a sample label, and an active contour map corresponding to the sample image. The sample label represents whether the category of the sample image is a defective image or a non-defective image. The active contour map corresponding to the defective image is an image marked with the packaging contour and the defect center position, and the active contour map corresponding to the non-defective image is an image marked with the packaging contour.

[0093] Optionally, in the embodiment of the present application, the defect detection model includes: an encoder and a classifier; the detection result acquisition module includes:

[0094] A mapping matrix acquisition module, configured to use the encoder to perform feature extraction on the to-be-detected image to obtain a feature mapping matrix.

[0095] A mapping matrix classification module, configured to use the classifier to classify the feature mapping matrix to obtain a defect detection result.

[0096] Optionally, in the embodiment of the present application, the defect detection model further includes: a decoder; the defect detection device further includes:

[0097] An active contour prediction module, configured to use a decoder to predict an active contour map corresponding to an image to be detected according to a feature mapping matrix.

[0098] Optionally, in an embodiment of the present application, the defect detection device further includes:

[0099] A sample data acquisition module, configured to acquire a sample image, a sample label, and an active contour map corresponding to the sample image.

[0100] A detection model acquisition module, configured to use the sample image and the active contour map corresponding to the sample image to train an encoder and a decoder, and use the sample image and the sample label to train the encoder and a classifier, and obtain a defect detection model by means of joint training.

[0101] Optionally, in an embodiment of the present application, the sample data acquisition module includes:

[0102] An image label acquisition module, configured to acquire a sample image and a sample label corresponding to the sample image.

[0103] A first image annotation module, configured to, if the sample label corresponding to the sample image is a defective image, annotate the packaging contour and the defective center position in the sample image to obtain a first annotated image, and generate an active contour map corresponding to the sample image according to the first annotated image.

[0104] A second image annotation module, configured to, if the sample label corresponding to the sample image is a non-defective image, annotate the packaging contour in the sample image to obtain a second annotated image, and generate an active contour map corresponding to the sample image according to the second annotated image.

[0105] Optionally, in an embodiment of the present application, the detection model acquisition module includes:

[0106] An image feature extraction module, configured to use an encoder to extract features from a sample image to obtain a feature mapping matrix corresponding to the sample image.

[0107] An image contour prediction module, configured to use a decoder to perform contour prediction on the feature mapping matrix corresponding to the sample image to obtain a predicted active contour map.

[0108] A first model training module, configured to calculate an active contour loss between the predicted active contour map and the active contour map corresponding to the sample image, and train the encoder and the decoder according to the active contour loss.

[0109] Optionally, in an embodiment of the present application, the detection model acquisition module further includes:

[0110] The image feature extraction module is used to use the encoder to extract features from the sample image and obtain a feature mapping matrix corresponding to the sample image.

[0111] The image category prediction module is used to use a classifier to perform category prediction on the feature mapping matrix corresponding to the sample image to obtain the predicted category of the sample image.

[0112] The second model training module is used to calculate the image classification loss between the predicted category of the sample image and the category of the sample label, and train the encoder and the classifier according to the image classification loss.

[0113] It should be understood that the device corresponds to the above-mentioned defect detection method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device.

[0114] An electronic device provided in an embodiment of the present application includes: a processor and a memory, the memory storing machine-readable instructions executable by the processor, and the above method is performed when the machine-readable instructions are executed by the processor.

[0115] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. Wherein, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk.

[0116] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the similarities between the various embodiments, reference can be made to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the corresponding descriptions in the method embodiments.

[0117] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may also occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, mainly depending on the functions involved.

[0118] In addition, in each of the embodiments of the present application, the various functional modules may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part. Furthermore, in the description of this specification, the descriptions of reference terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0119] In this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0120] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the embodiments of the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the embodiments of the present application.

Claims

1. A defect detection method, characterized in that, Including: Obtain an image to be detected, where the image to be detected is obtained by photographing the packaging of a target item; Use a defect detection model to perform defect detection on the image to be detected to obtain a defect detection result. The defect detection model is obtained by training using sample images, sample labels, and the active contour maps corresponding to the sample images. The sample labels represent whether the category of the sample image is a defective image or a non-defective image. The active contour map corresponding to the defective image is an image marked with the packaging contour and the defect center position, and the active contour map corresponding to the non-defective image is an image marked with the packaging contour; Wherein, before using the defect detection model to perform defect detection on the image to be detected, it further includes: obtaining sample images, sample labels, and the active contour maps corresponding to the sample images; using the sample images and the active contour maps corresponding to the sample images to train an encoder and a decoder, and using the sample images and the sample labels to train the encoder and a classifier, and obtaining the defect detection model through a joint training method; The obtaining of the sample images, sample labels, and the active contour maps corresponding to the sample images includes: obtaining sample images and the sample labels corresponding to the sample images; if the sample label corresponding to the sample image is a defective image, then mark the packaging contour and the defect center position in the sample image to obtain a first marked image, and generate the active contour map corresponding to the sample image according to the first marked image; if the sample label corresponding to the sample image is a non-defective image, then mark the packaging contour in the sample image to obtain a second marked image, and generate the active contour map corresponding to the sample image according to the second marked image. The active contour map corresponding to the sample image is used to provide a spatial attention mechanism for the defect detection model.

2. The method according to claim 1, wherein The defect detection model includes: an encoder and a classifier; the using of the defect detection model to perform defect detection on the image to be detected to obtain a defect detection result includes: Using the encoder to perform feature extraction on the image to be detected to obtain a feature mapping matrix; Using the classifier to classify the feature mapping matrix to obtain the defect detection result.

3. The method according to claim 2, wherein The defect detection model further includes: a decoder; after obtaining the feature mapping matrix, it further includes: Using the decoder to predict the active contour map corresponding to the image to be detected according to the feature mapping matrix.

4. The method according to claim 1, characterized in that, The using of the sample images and the active contour maps corresponding to the sample images to train the encoder and the decoder includes: Using the encoder to perform feature extraction on the sample image to obtain the feature mapping matrix corresponding to the sample image; Using the decoder to perform contour prediction on the feature mapping matrix corresponding to the sample image to obtain the predicted active contour map; Calculating the active contour loss between the predicted active contour map and the active contour map corresponding to the sample image, and training the encoder and the decoder according to the active contour loss.

5. The method according to claim 1, wherein Training the encoder and the classifier using the sample image and the sample label includes: Using the encoder to extract features from the sample image to obtain a feature mapping matrix corresponding to the sample image; Using the classifier to perform class prediction on the feature mapping matrix corresponding to the sample image to obtain a predicted class of the sample image; Calculating an image classification loss between the predicted class of the sample image and the class of the sample label, and training the encoder and the classifier according to the image classification loss.

6. A defect detection device, characterized in that, Including: A detection image acquisition module for acquiring a to-be-detected image, where the to-be-detected image is obtained by photographing a target item package; A detection result acquisition module for using a defect detection model to perform defect detection on the to-be-detected image to obtain a defect detection result, where the defect detection model is obtained by training using a sample image, a sample label, and an active contour map corresponding to the sample image, the sample label characterizes that the class of the sample image is a defective image or a non-defective image, the active contour map corresponding to the defective image is an image marked with a package contour and a defect center position, and the active contour map corresponding to the non-defective image is an image marked with a package contour; Wherein, before using the defect detection model to perform defect detection on the to-be-detected image, it further includes: acquiring a sample image, a sample label, and an active contour map corresponding to the sample image; using the sample image and the active contour map corresponding to the sample image to train an encoder and a decoder, and using the sample image and the sample label to train the encoder and a classifier, and obtaining the defect detection model through a joint training method; The acquiring the sample image, the sample label, and the active contour map corresponding to the sample image includes: acquiring a sample image and a sample label corresponding to the sample image; if the sample label corresponding to the sample image is a defective image, then marking the package contour and the defect center position in the sample image to obtain a first marked image, and generating the active contour map corresponding to the sample image according to the first marked image; if the sample label corresponding to the sample image is a non-defective image, then marking the package contour in the sample image to obtain a second marked image, and generating the active contour map corresponding to the sample image according to the second marked image, and the active contour map corresponding to the sample image is used to provide a spatial attention mechanism for the defect detection model.

7. An electronic device, characterized in that, Including: A processor and a memory, where the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is executed.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the method according to any one of claims 1 to 5 is executed.

Images