Product surface defect detection method and device, electronic equipment and storage medium
By jointly training semantic segmentation and super-resolution models, the problem of low accuracy and efficiency of product surface defect detection in the prior art is solved, efficient and accurate defect detection is achieved, and missed detection rate is reduced and calculation cost is reduced.
Patent Information
- Application Number
- CN202310485874.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-05
AI Technical Summary
The existing product surface defect detection methods have problems such as low accuracy and low model mobility. This is mainly due to the low image resolution and the dependence of high-resolution images on defect detection, resulting in low detection efficiency and high missed detection rate.
By jointly training the initial semantic segmentation model and super-resolution model, the image semantic segmentation model and super-resolution model are generated, the feature similarity loss is calculated using semantic segmentation feature maps and super-resolution feature maps, and the model parameters are iteratively updated to achieve low resolution Efficient defect detection of rate images.
Improves the accuracy and efficiency of product surface defect detection, reduces leakage detection rate, reduces dependence on high-resolution images without increasing computing costs.
Smart Images

Figure CN118898566A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of defect detection technology, and in particular to a product surface defect detection method, device, electronic equipment and storage medium. Background Art
[0002] In the production and use scenarios of traditional industrial manufacturing, quality inspection is a key link. For example, in the fields of refrigerators, televisions, automobiles, printed circuit boards, chips and liquid crystal displays, testing the surface condition of products is an important means of controlling product quality. In addition, many important equipment are exposed to the outside for a long time and are affected by climatic conditions, which easily cause rust and other destructive phenomena on the surface of their components, resulting in equipment damage and even safety accidents in serious cases. For example, rust on power transmission line equipment and rust on oil storage tanks. Therefore, product quality inspection is very important.
[0003] The current quality inspection methods include manual inspection and machine inspection. Manual inspection requires a lot of manpower and time costs, with low inspection efficiency and high missed inspection rate. Machine inspection mainly uses drones or robots to take pictures of products, obtain images of the current status of the products, and use image processing technology to automatically diagnose the products.
[0004] However, due to practical problems such as low resolution of captured images, existing defect detection methods have problems such as low accuracy and low model transferability. Summary of the invention
[0005] The present invention provides a product surface defect detection method, device, electronic equipment and storage medium, which are used to solve the defects existing in the prior art.
[0006] The present invention provides a method for detecting surface defects of a product, comprising:
[0007] Acquire a surface image of the product to be inspected;
[0008] Inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model;
[0009] The defect detection model is trained based on the following steps:
[0010] Based on the training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image;
[0011] Based on the test samples, the image semantic segmentation model is tested to obtain the defect detection model; the training samples and the test samples both include sample images of sample product surfaces and defect labels of the sample product surfaces.
[0012] According to a product surface defect detection method provided by the present invention, based on training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model, including:
[0013] Inputting sample images of the sample product surface in the training samples into the initial semantic segmentation model and the initial super-resolution model respectively, obtaining a semantic segmentation feature map extracted by the initial semantic segmentation model and a category prediction probability map output by the initial semantic segmentation model based on the semantic segmentation feature map, obtaining a super-resolution feature map extracted by the initial super-resolution model and a super-resolution prediction map output by the initial super-resolution model based on the super-resolution feature map;
[0014] Based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map, the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model is calculated, based on the category prediction probability map and the defect labels of the sample images of the sample product surfaces in the training samples, the semantic segmentation loss is calculated, and based on the super-resolution prediction map and the super-resolution standard map corresponding to the sample images of the sample product surfaces in the training samples, the super-resolution loss is calculated;
[0015] Based on the feature similarity loss, the semantic segmentation loss and the super-resolution loss, the structural parameters of the initial semantic segmentation model and the initial super-resolution model are iteratively updated to obtain the image semantic segmentation model and the image super-resolution model.
[0016] According to a product surface defect detection method provided by the present invention, the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model is calculated based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map, including:
[0017] Based on the similarity between every two pixels in the semantic segmentation feature map, determining a first similarity matrix corresponding to the semantic segmentation feature map, and based on the similarity between every two pixels in the super-resolution feature map, determining a second similarity matrix corresponding to the super-resolution feature map;
[0018] The feature similarity loss is calculated based on similarity parameter values at the same position in the first similarity matrix and the second similarity matrix.
[0019] According to a product surface defect detection method provided by the present invention, the initial semantic segmentation model and the initial super-resolution model both include a coding structure and a decoding structure, the coding structure and the decoding structure in the initial semantic segmentation model are used to extract the semantic segmentation feature map, and the coding structure and the decoding structure in the initial super-resolution model are used to extract the super-resolution feature map;
[0020] The encoding structure of the initial semantic segmentation model is the same as that of the initial super-resolution model and they share structural parameters.
[0021] According to a product surface defect detection method provided by the present invention, the super-resolution standard image is a sample image in which the resolution of the sample product surface in the training sample is higher than a first preset threshold, or is obtained by upsampling a sample image in which the resolution of the sample product surface in the training sample is lower than a second preset threshold.
[0022] According to a product surface defect detection method provided by the present invention, the surface image is input into a defect detection model to obtain a detection result output by the defect detection model, including:
[0023] Performing image enhancement processing on the surface image to obtain a processing result;
[0024] The processing result is input into a defect detection model to obtain the detection result.
[0025] The present invention also provides a product surface defect detection device, comprising:
[0026] An image acquisition module, used to acquire a surface image of the product to be inspected;
[0027] A defect detection module, used for inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model;
[0028] The model training module is used to train the defect detection model based on the following steps:
[0029] Based on the training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image;
[0030] Based on the test samples, the image semantic segmentation model is tested to obtain the defect detection model; the training samples and the test samples both include sample images of sample product surfaces and defect labels of the sample product surfaces.
[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a product surface defect detection method as described in any one of the above is implemented.
[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the product surface defect detection method as described in any one of the above is implemented.
[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the product surface defect detection method as described in any one of the above is implemented.
[0034] The present invention provides a product surface defect detection method, device, electronic device and storage medium. The method first obtains the surface image of the product to be detected; then uses the defect detection model to determine the detection result. In this method, the initial semantic segmentation model and the initial super-resolution model are jointly trained using training samples, and the trained image semantic segmentation model is tested with test samples to obtain a defect detection model, so that intelligent detection of product surface defects can be achieved. Compared with the traditional method of manual inspection, the detection result is more accurate and the detection efficiency is higher. Moreover, the method uses semantic segmentation to achieve pixel-level classification detection and effectively reduce the missed detection rate. In addition, the method also effectively solves the reliance of existing machine inspection methods on high-resolution input images, while effectively improving the segmentation efficiency, reducing the amount of network calculations without adding additional computing costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without paying any creative work.
[0036] Figure 1 This is one of the flow charts of the product surface defect detection method provided by the present invention;
[0037] Figure 2 It is a schematic diagram of the training process of the defect detection model in the product surface defect detection method provided by the present invention;
[0038] Figure 3 This is the second flow chart of the product surface defect detection method provided by the present invention;
[0039] Figure 4 It is a structural schematic diagram of a product surface defect detection device provided by the present invention;
[0040] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] The existing defect detection methods have technical problems such as low accuracy and low model transferability due to practical problems such as limited number of captured images, low image resolution, and no significant regularity of product surface defects. To this end, an embodiment of the present invention provides a product surface defect detection method to solve the above technical problems.
[0043] Figure 1 FIG. 1 is a flow chart of a method for detecting surface defects of a product provided in an embodiment of the present invention. Figure 1 As shown, the method includes:
[0044] S1, obtaining a surface image of the product to be inspected;
[0045] S2, inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model;
[0046] The defect detection model is trained based on the following steps:
[0047] Based on the training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image;
[0048] Based on the test samples, the image semantic segmentation model is tested to obtain the defect detection model; the training samples and the test samples both include sample images of sample product surfaces and defect labels of the sample product surfaces.
[0049] Specifically, the product surface defect detection method provided in the embodiment of the present invention is executed by a product surface defect detection device, which can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.
[0050] First, step S1 is performed to obtain a surface image of a product to be inspected. The product to be inspected may be a product whose surface needs to be inspected for defects, such as industrial products such as equipment, components, and parts. The surface image of the product to be inspected may be acquired by an image acquisition device connected to a product surface defect detection device and transmitted to the product surface defect detection device.
[0051] The image acquisition device may be a camera carried by a drone or a robot, and the surface image of the product to be inspected may be obtained by photographing the surface of the product to be inspected at different angles. For example, the surface image of the product to be inspected may be obtained by photographing the surface of the product to be inspected at 360 degrees. The surface image may be a color image or a grayscale image, which is not specifically limited here.
[0052] Then, step S2 is executed to perform semantic segmentation on the surface image with the help of the defect detection model to obtain the detection result output by the defect detection model. The detection result can be the classification result of each pixel in the surface image, that is, the detection result can be a classification map containing the classification result of each pixel in the surface image. The classification result can include defective or non-defective. When there is a defect, the defect category can be rust or cracks, that is, the product surface defect detection method can be used to detect the presence or absence of defects such as rust and cracks on the surface of the product.
[0053] After the test result is obtained, the test result can also be displayed in the surface image of the product to be tested. For example, if the test result shows that a certain position in the surface image is defective, the position can be framed with a rectangular frame, and the defect category can be displayed in the rectangular frame.
[0054] In the process of training the defect detection model, the initial semantic segmentation model (Semantic Segmentation Module, SSM) and the initial super resolution model (Super Resolution Module, SRM) are first jointly trained using the training samples to obtain the image semantic segmentation model and the image super resolution model. The number of training samples can be set as needed and can include multiple samples, which is not specifically limited here. The training samples can include sample images of the sample product surface and defect labels of the sample product surface, and the defect labels can be determined by manual annotation.
[0055] The initial semantic segmentation model and the initial super-resolution model can be jointly supervised and trained. When conducting joint supervised training, the inputs of the initial semantic segmentation model and the initial super-resolution model are sample images of the sample product surface in the training samples. The model loss used can be determined based on the output results of the initial semantic segmentation model and the defect labels of the sample product surface in the training samples, and the output results of the initial super-resolution model and the super-resolution standard image corresponding to the training samples. The end of training can be marked by the convergence of the model loss.
[0056] After training, the initial semantic segmentation model can obtain multiple image semantic segmentation models. The image semantic segmentation model has a semantic segmentation function and is used to perform semantic segmentation on the input image and realize pixel-level classification of the input image. After training, the initial super-resolution model can obtain multiple image super-resolution models. The image super-resolution model can effectively help the image semantic segmentation model learn the super-resolution feature representation of the input image, make up for the defect of insufficient resolution of the input image, reduce the dependence of the image semantic segmentation model on high-resolution images, and help improve the learning effect of the image semantic segmentation model. Here, the input image is not specific, but is only used to characterize the functions of the image semantic segmentation model and the image super-resolution model obtained by joint training.
[0057] Afterwards, the image super-resolution model is discarded, and each image semantic segmentation model is tested using the test samples to verify the generalization ability of each image semantic segmentation model, determine the semantic segmentation accuracy of each image semantic segmentation model for the test samples, and select the image semantic segmentation model with the highest accuracy as the defect detection model. Here, the number of test samples can be set as needed and can include multiple ones, which is not specifically limited here. The test samples may also include sample images of the surface of the sample product and defect labels on the surface of the sample product, and the defect labels may also be determined by manual annotation. The measure of semantic segmentation accuracy may be the proportion of correct semantic segmentation of all test samples by each image semantic segmentation model.
[0058] The product surface defect detection method provided in the embodiment of the present invention first obtains the surface image of the product to be detected; then uses the defect detection model to determine the detection result. In this method, the initial semantic segmentation model and the initial super-resolution model are jointly trained using training samples, and the trained image semantic segmentation model is tested with test samples to obtain a defect detection model, so that intelligent detection of product surface defects can be achieved. Compared with the traditional method of manual inspection, the detection result is more accurate and the detection efficiency is higher. Moreover, the method uses semantic segmentation to achieve pixel-level classification detection, effectively reducing the missed detection rate. In addition, the method also effectively solves the reliance of existing machine inspection methods on high-resolution input images, while effectively improving the segmentation efficiency, reducing the amount of network calculations without adding additional computing costs.
[0059] On the basis of the above-mentioned embodiment, the product surface defect detection method provided in the embodiment of the present invention, wherein the surface image is input into the defect detection model to obtain the detection result output by the defect detection model, comprises:
[0060] Performing image enhancement processing on the surface image to obtain a processing result;
[0061] The processing result is input into a defect detection model to obtain the detection result.
[0062] Specifically, before the surface image is input into the defect detection model, the surface image may be first subjected to image enhancement processing to obtain a processing result. The image enhancement processing may include operations such as rotation, translation, brightness change, etc., to achieve data enhancement of the surface image.
[0063] Thereafter, the processing results may be input into a defect detection model, and the defect detection model may be used to perform semantic segmentation on the processing results pixel by pixel to obtain the final detection result.
[0064] Accordingly, in the process of training the defect detection model, the sample images of the sample product surfaces in the training samples and the test samples also need to be enhanced first, and then the processed results are subsequently applied.
[0065] In the embodiment of the present invention, a step of performing data enhancement processing on the surface image is introduced to make the obtained detection result more accurate.
[0066] On the basis of the above embodiments, the product surface defect detection method provided in the embodiments of the present invention, based on the training samples, jointly trains the initial semantic segmentation model and the initial super-resolution model to obtain the image semantic segmentation model and the image super-resolution model, including:
[0067] Inputting sample images of the sample product surface in the training samples into the initial semantic segmentation model and the initial super-resolution model respectively, obtaining a semantic segmentation feature map extracted by the initial semantic segmentation model and a category prediction probability map output by the initial semantic segmentation model based on the semantic segmentation feature map, obtaining a super-resolution feature map extracted by the initial super-resolution model and a super-resolution prediction map output by the initial super-resolution model based on the super-resolution feature map;
[0068] Based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map, the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model is calculated, based on the category prediction probability map and the defect labels of the sample images of the sample product surfaces in the training samples, the semantic segmentation loss is calculated, and based on the super-resolution prediction map and the super-resolution standard map corresponding to the sample images of the sample product surfaces in the training samples, the super-resolution loss is calculated;
[0069] Based on the feature similarity loss, the semantic segmentation loss and the super-resolution loss, the structural parameters of the initial semantic segmentation model and the initial super-resolution model are iteratively updated to obtain the image semantic segmentation model and the image super-resolution model.
[0070] Specifically, when jointly training the initial semantic segmentation model and the initial super-resolution model using training samples, sample images of the sample product surface in the training samples may be first input into the initial semantic segmentation model and the initial super-resolution model, respectively.
[0071] The initial semantic segmentation model and the initial super-resolution model encode and decode the input sample image respectively to obtain a semantic segmentation feature map F1 extracted by the initial semantic segmentation model and a super-resolution feature map F2 extracted by the initial super-resolution model.
[0072] Assuming that the size of the input image is H×W×C1, the size of the semantic segmentation feature map F1 is 2H×2W×C2. However, the semantic segmentation feature map F1 obtained by relying solely on the decoding unit of the initial semantic segmentation model is not sufficient to be a good image super-resolution feature representation. Therefore, the initial super-resolution model is introduced, and the generated super-resolution feature map F2 is the same size as the semantic segmentation feature map F1, which is used to help the initial semantic segmentation model learn the super-resolution features of the input image. The number of pixels in the semantic segmentation feature map F1 and the super-resolution feature map F2 is the same. The pixel value at each pixel in the semantic segmentation feature map F1 and the super-resolution feature map F2 is a multi-dimensional feature vector. The feature vector in the semantic segmentation feature map F1 represents the semantic segmentation feature vector at the pixel point, and the feature vector in the super-resolution feature map F2 represents the super-resolution feature vector at the pixel point.
[0073] Thereafter, the initial semantic segmentation model can further use the semantic segmentation feature map F1 to obtain and output a category prediction probability map. For example, the initial semantic segmentation model can be configured with a classification layer, and the semantic segmentation feature map F1 is processed using a softmax function to obtain a category prediction probability map.
[0074] The size of the class prediction probability map is the same as the size of the input sample image, and the class prediction probability map includes the class prediction probability value of each pixel in the input sample image. In the embodiment of the present invention, the number of categories may include multiple, the class prediction probability value of each pixel may be the maximum value of the class prediction probability values of the pixel belonging to each category, and the number of pixels in the class prediction probability map is the same as the number of pixels in the input sample image.
[0075] The initial super-resolution model can further utilize the super-resolution feature map F2 to output a super-resolution prediction map.
[0076] After that, the total loss required in the model training process is calculated, which can be composed of three parts: feature similarity loss L fs , semantic segmentation loss L ss And the super-resolution loss L SR . Feature similarity loss L fs The goal is to use the super-resolution feature map generated by the initial super-resolution model to guide the learning of the initial semantic segmentation model, thereby making up for the defect of insufficient resolution of the input image and reducing the dependence of the initial semantic segmentation model on high-resolution input images. SS It is used to learn the structural parameters of the initial semantic segmentation model to help complete the pixel-by-pixel surface defect classification and achieve the purpose of surface defect recognition. SR It is used to learn the structural parameters of the initial super-resolution model so that the initial super-resolution model has the function of generating a super-resolution prediction map for a low-resolution input image.
[0077] For feature similarity loss L fs The feature similarity loss L between the initial semantic segmentation model and the initial super-resolution model can be calculated by using the similarity between every two pixels in the semantic segmentation feature map F1 and the similarity between every two pixels in the super-resolution feature map F2. fs .
[0078] The similarity between every two pixels in the semantic segmentation feature map F1 can be characterized by parameters such as the dot product operation result, Euclidean distance, cosine distance, etc. between the semantic segmentation feature vectors at every two pixels. The similarity between every two pixels in the super-resolution feature map F2 can be characterized by similarity parameters such as the dot product operation result, Euclidean distance, cosine distance, etc. between the super-resolution feature vectors at every two pixels.
[0079] The feature similarity loss L can be determined by using the difference in similarity between two pixels at the same position in the semantic segmentation feature map F1 and the super-resolution feature map F2. fs .
[0080] For semantic segmentation loss L Ss , then the defect labels of sample images of sample product surfaces in the training samples can be calculated using the cross entropy loss function. The calculation formula can be:
[0081]
[0082] Among them, p i Represents the category prediction probability value of pixel i in the sample image, y i represents the category label of pixel i in the sample image, and N1 represents the total number of pixels in the sample image.
[0083] For the super-resolution loss L SR , using the super-resolution prediction map and the super-resolution standard map corresponding to the sample image of the sample product surface in the training sample, the mean square error loss function (MSE) is used for calculation. The calculation formula can be:
[0084]
[0085] Among them, X i represents the pixel value at pixel i in the super-resolution prediction image, Y i represents the pixel value at pixel i in the super-resolution standard image, and N2 represents the total number of pixels in the super-resolution prediction image.
[0086] Finally, using the feature similarity loss L fs , semantic segmentation loss L SS And the super-resolution loss L SR , calculate the total loss, and use the total loss to iteratively update the structural parameters of the initial semantic segmentation model and the initial super-resolution model to obtain the image semantic segmentation model and the image super-resolution model.
[0087] In the embodiment of the present invention, three losses are introduced, which can free the final image semantic segmentation model from the limitation of insufficient resolution of the input image, improve its semantic segmentation effect, and make the final detection result more accurate.
[0088] On the basis of the above embodiment, the product surface defect detection method provided in the embodiment of the present invention, based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map, calculates the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model, including:
[0089] Based on the similarity between every two pixels in the semantic segmentation feature map, determining a first similarity matrix corresponding to the semantic segmentation feature map, and based on the similarity between every two pixels in the super-resolution feature map, determining a second similarity matrix corresponding to the super-resolution feature map;
[0090] The feature similarity loss is calculated based on similarity parameter values at the same position in the first similarity matrix and the second similarity matrix.
[0091] Specifically, the feature similarity loss L fs In the calculation process, the similarity between every two pixels in the semantic segmentation feature map F1 can be used to determine the first similarity matrix D1 corresponding to the semantic segmentation feature map F1. The elements at each position in the first similarity matrix D1 are the similarity parameters between two pixels in the semantic segmentation feature map F1.
[0092] The second similarity matrix D2 corresponding to the super-resolution feature map F2 is determined by using the similarity between every two pixels in the super-resolution feature map F2. The elements at each position in the second similarity matrix D2 are the similarity parameters between two pixels in the super-resolution feature map F2.
[0093] Finally, the feature similarity loss L is calculated by using the similarity parameter values at the same position in the first similarity matrix D1 and the second similarity matrix D2. fs The calculation formula is as follows:
[0094]
[0095] Wherein, W is the number of pixels in the width direction of the semantic segmentation feature map F1 and the super-resolution feature map F2, and H is the number of pixels in the height direction of the semantic segmentation feature map F1 and the super-resolution feature map F2. represents an element at a certain position in the first similarity matrix D1, representing the similarity parameter value between the i-th and j-th pixel points in the semantic segmentation feature map F1, The element at a certain position in the second similarity matrix D2 represents the value of the similarity parameter between the i-th and j-th pixel points in the super-resolution feature map F2.
[0096] In the embodiment of the present invention, when calculating the feature similarity loss, the similarity between any two pixels in the feature map is taken into account, so that the data used in calculating the feature similarity loss can be more complete and the calculation result can be more accurate.
[0097] On the basis of the above embodiment, in the product surface defect detection method provided in the embodiment of the present invention, the initial semantic segmentation model and the initial super-resolution model both include a coding structure and a decoding structure, the coding structure and the decoding structure in the initial semantic segmentation model are used to extract the semantic segmentation feature map, and the coding structure and the decoding structure in the initial super-resolution model are used to extract the super-resolution feature map;
[0098] The encoding structure of the initial semantic segmentation model is the same as that of the initial super-resolution model and they share structural parameters.
[0099] Specifically, in an embodiment of the present invention, the initial semantic segmentation model and the initial super-resolution model both include an encoding (Encoder) structure and a decoding (Decoder) structure. The encoding structure and decoding structure in the initial semantic segmentation model are used to extract semantic segmentation feature maps. The encoding structure in the initial semantic segmentation model can be a convolutional neural network structure, and the decoding structure can be an upsampling structure.
[0100] The encoding structure and decoding structure in the initial super-resolution model are used to extract super-resolution feature maps. The encoding structure in the initial super-resolution model can also be a convolutional neural network structure, which can be the same as the encoding structure in the initial semantic segmentation model and share structural parameters. The decoding structure in the initial super-resolution model can also be an upsampling structure, which can be different from the decoding structure in the initial semantic segmentation model.
[0101] In an embodiment of the present invention, setting the encoding structure in the initial semantic segmentation model and the initial super-resolution model to be the same and sharing structural parameters can simplify the structural complexity of the initial semantic segmentation model and the initial super-resolution model and improve learning efficiency.
[0102] On the basis of the above embodiments, in the product surface defect detection method provided in the embodiments of the present invention, the super-resolution standard image is a sample image in which the resolution of the sample product surface in the training sample is higher than the first preset threshold, or is obtained based on upsampling the sample image in which the resolution of the sample product surface in the training sample is lower than the second preset threshold.
[0103] Specifically, in the embodiment of the present invention, a sample image with a resolution higher than a first preset threshold is a high-resolution sample image, and a sample image with a resolution lower than a second preset threshold is a low-resolution sample image. Here, the first preset threshold and the second preset threshold can be set as needed and are not specifically limited here.
[0104] For sample products with high-resolution sample images available, the high-resolution sample images of the sample products are used as super-resolution standard images, and the low-resolution sample images in the training samples are obtained by downsampling the super-resolution standard images and introducing noise.
[0105] For sample products where it is difficult to obtain high-resolution sample images, the super-resolution standard image is obtained by upsampling the low-resolution sample images in the training samples. This can not only effectively ensure the number of sample images, but also improve the good performance of the trained image semantic segmentation model.
[0106] like Figure 2 As shown, based on the above embodiments, the product surface defect detection method provided in the embodiments of the present invention, the defect detection model training process may include:
[0107] The sample images are respectively input into the initial semantic segmentation model and the initial super-resolution model. The semantic segmentation feature map is extracted through the encoding structure and decoding structure of the initial semantic segmentation model, and the category prediction probability map is output through the classification layer. The super-resolution feature map is extracted through the encoding structure and decoding structure of the initial super-resolution model, and the super-resolution prediction map is output.
[0108] After that, the feature similarity loss is calculated using the semantic segmentation feature map and the super-resolution feature map, the semantic segmentation loss is calculated using the category prediction probability map and the category label, and the super-resolution loss is calculated using the super-resolution prediction map and the super-resolution standard map. Then, the feature similarity loss, semantic segmentation loss and super-resolution loss are summarized to obtain the total loss.
[0109] The total loss is used to iteratively update the structural parameters of the initial semantic segmentation model and the initial super-resolution model to obtain the image semantic segmentation model and the image super-resolution model.
[0110] Finally, the image semantic segmentation model is tested using the test samples to obtain the final defect detection model.
[0111] like Figure 3 As shown, based on the above embodiment, the product surface defect detection method provided in the embodiment of the present invention includes:
[0112] The surface image of the product to be inspected is input into the defect detection model. The encoding structure and decoding structure in the defect detection model perform semantic segmentation feature extraction on the surface image, and the detection result is obtained through the classification layer.
[0113] like Figure 4 As shown, based on the above embodiment, an embodiment of the present invention provides a product surface defect detection device, including:
[0114] An image acquisition module 41 is used to acquire a surface image of a product to be inspected;
[0115] A defect detection module 42, used for inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model;
[0116] The model training module 43 is used to train the defect detection model based on the following steps:
[0117] Based on the training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image;
[0118] Based on the test samples, the image semantic segmentation model is tested to obtain the defect detection model; the training samples and the test samples both include sample images of sample product surfaces and defect labels of the sample product surfaces.
[0119] On the basis of the above-mentioned embodiments, in the product surface defect detection device provided in the embodiments of the present invention, the model training module is specifically used for:
[0120] Inputting sample images of the sample product surface in the training samples into the initial semantic segmentation model and the initial super-resolution model respectively, obtaining a semantic segmentation feature map extracted by the initial semantic segmentation model and a category prediction probability map output by the initial semantic segmentation model based on the semantic segmentation feature map, obtaining a super-resolution feature map extracted by the initial super-resolution model and a super-resolution prediction map output by the initial super-resolution model based on the super-resolution feature map;
[0121] Based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map, the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model is calculated, based on the category prediction probability map and the defect labels of the sample images of the sample product surfaces in the training samples, the semantic segmentation loss is calculated, and based on the super-resolution prediction map and the super-resolution standard map corresponding to the sample images of the sample product surfaces in the training samples, the super-resolution loss is calculated;
[0122] Based on the feature similarity loss, the semantic segmentation loss and the super-resolution loss, the structural parameters of the initial semantic segmentation model and the initial super-resolution model are iteratively updated to obtain the image semantic segmentation model and the image super-resolution model.
[0123] On the basis of the above-mentioned embodiment, in the product surface defect detection device provided in the embodiment of the present invention, the model training module is further specifically used for:
[0124] Based on the similarity between every two pixels in the semantic segmentation feature map, determining a first similarity matrix corresponding to the semantic segmentation feature map, and based on the similarity between every two pixels in the super-resolution feature map, determining a second similarity matrix corresponding to the super-resolution feature map;
[0125] The feature similarity loss is calculated based on similarity parameter values at the same position in the first similarity matrix and the second similarity matrix.
[0126] On the basis of the above-mentioned embodiment, in the product surface defect detection device provided in the embodiment of the present invention, the initial semantic segmentation model and the initial super-resolution model both include a coding structure and a decoding structure, the coding structure and the decoding structure in the initial semantic segmentation model are used to extract the semantic segmentation feature map, and the coding structure and the decoding structure in the initial super-resolution model are used to extract the super-resolution feature map;
[0127] The encoding structure of the initial semantic segmentation model is the same as that of the initial super-resolution model and they share structural parameters.
[0128] Based on the above embodiments, in the product surface defect detection device provided in the embodiments of the present invention, the super-resolution standard image is a sample image in which the resolution of the sample product surface in the training sample is higher than the first preset threshold, or is obtained by upsampling the sample image in which the resolution of the sample product surface in the training sample is lower than the second preset threshold.
[0129] On the basis of the above-mentioned embodiment, in the product surface defect detection device provided in the embodiment of the present invention, the defect detection module is specifically used for:
[0130] Performing image enhancement processing on the surface image to obtain a processing result;
[0131] The processing result is input into a defect detection model to obtain the detection result.
[0132] Specifically, the functions of each module in the product surface defect detection device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.
[0133] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor (Processor) 510, a communication interface (Communications Interface) 520, a memory (Memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 complete mutual communication through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the product surface defect detection method provided in the above embodiments, the method comprising: obtaining a surface image of the product to be detected; inputting the surface image into the defect detection model to obtain the detection result output by the defect detection model; wherein the defect detection model is trained based on the following steps: based on the training sample, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to semantically segment the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image; based on the test sample, the image semantic segmentation model is tested to obtain the defect detection model; the training sample and the test sample both include a sample image of the sample product surface and a defect label of the sample product surface.
[0134] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.
[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the product surface defect detection method provided in the above-mentioned embodiments, the method including: obtaining a surface image of the product to be inspected; inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model; wherein the defect detection model is trained based on the following steps: based on training samples, an initial semantic segmentation model and an initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image; based on a test sample, the image semantic segmentation model is tested to obtain the defect detection model; the training sample and the test sample both include a sample image of the sample product surface and a defect label of the sample product surface.
[0136] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the product surface defect detection method provided in the above-mentioned embodiments, the method comprising: obtaining a surface image of the product to be inspected; inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model; wherein the defect detection model is trained based on the following steps: based on training samples, jointly training an initial semantic segmentation model and an initial super-resolution model to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image; based on a test sample, the image semantic segmentation model is tested to obtain the defect detection model; the training sample and the test sample both include a sample image of the sample product surface and a defect label of the sample product surface.
[0137] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for detecting surface defects of a product, characterized in that: include: Acquire a surface image of the product to be inspected; Inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model; The defect detection model is trained based on the following steps: Based on the training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image; Based on the test samples, the image semantic segmentation model is tested to obtain the defect detection model; the training samples and the test samples both include sample images of sample product surfaces and defect labels of the sample product surfaces.
2. The product surface defect detection method according to claim 1, characterized in that: The method of jointly training the initial semantic segmentation model and the initial super-resolution model based on the training samples to obtain the image semantic segmentation model and the image super-resolution model includes: Inputting sample images of the sample product surface in the training samples into the initial semantic segmentation model and the initial super-resolution model respectively, obtaining a semantic segmentation feature map extracted by the initial semantic segmentation model and a category prediction probability map output by the initial semantic segmentation model based on the semantic segmentation feature map, obtaining a super-resolution feature map extracted by the initial super-resolution model and a super-resolution prediction map output by the initial super-resolution model based on the super-resolution feature map; Based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map, the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model is calculated, based on the category prediction probability map and the defect labels of the sample images of the sample product surfaces in the training samples, the semantic segmentation loss is calculated, and based on the super-resolution prediction map and the super-resolution standard map corresponding to the sample images of the sample product surfaces in the training samples, the super-resolution loss is calculated; Based on the feature similarity loss, the semantic segmentation loss and the super-resolution loss, the structural parameters of the initial semantic segmentation model and the initial super-resolution model are iteratively updated to obtain the image semantic segmentation model and the image super-resolution model.
3. The product surface defect detection method according to claim 2, characterized in that: The calculating the feature similarity loss between the initial semantic segmentation model and the initial super-resolution model based on the similarity between every two pixels in the semantic segmentation feature map and the similarity between every two pixels in the super-resolution feature map includes: Based on the similarity between every two pixels in the semantic segmentation feature map, determining a first similarity matrix corresponding to the semantic segmentation feature map, and based on the similarity between every two pixels in the super-resolution feature map, determining a second similarity matrix corresponding to the super-resolution feature map; The feature similarity loss is calculated based on similarity parameter values at the same position in the first similarity matrix and the second similarity matrix.
4. The product surface defect detection method according to claim 2, characterized in that: The initial semantic segmentation model and the initial super-resolution model both include a coding structure and a decoding structure, the coding structure and the decoding structure in the initial semantic segmentation model are used to extract the semantic segmentation feature map, and the coding structure and the decoding structure in the initial super-resolution model are used to extract the super-resolution feature map; The encoding structure of the initial semantic segmentation model is the same as that of the initial super-resolution model and they share structural parameters.
5. The product surface defect detection method according to claim 2, characterized in that: The super-resolution standard image is a sample image in which the resolution of the sample product surface in the training sample is higher than a first preset threshold, or is obtained by upsampling a sample image in which the resolution of the sample product surface in the training sample is lower than a second preset threshold.
6. The method for detecting product surface defects according to any one of claims 1 to 5, characterized in that: The step of inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model includes: Performing image enhancement processing on the surface image to obtain a processing result; The processing result is input into a defect detection model to obtain the detection result.
7. A product surface defect detection device, characterized in that: include: An image acquisition module, used to acquire a surface image of the product to be inspected; A defect detection module, used for inputting the surface image into a defect detection model to obtain a detection result output by the defect detection model; The model training module is used to train the defect detection model based on the following steps: Based on the training samples, the initial semantic segmentation model and the initial super-resolution model are jointly trained to obtain an image semantic segmentation model and an image super-resolution model; the image semantic segmentation model is used to perform semantic segmentation on the input image, and the image super-resolution model is used to learn the super-resolution feature representation of the input image; Based on the test samples, the image semantic segmentation model is tested to obtain the defect detection model; the training samples and the test samples both include sample images of sample product surfaces and defect labels of the sample product surfaces.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the product surface defect detection method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the product surface defect detection method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the product surface defect detection method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Surface defect detection method and device based on joint model
CN113344888A
Remote sensing image semantic segmentation method for collaborative image super-resolution reconstruction
CN113436198A
Crack online detection method and system based on mobile device and super-resolution reconstruction segmentation network
CN115760720A
Target object detection method and apparatus, electronic device and storage medium
WO2022160597A1
