Product defect detection method, electronic device, storage medium, and program product

Abstract

The application provides a product defect detection method, an electronic device, a storage medium and a program product, and the method comprises the steps of determining a to-be-tested product image; and obtaining a product defect detection result, inputting the to-be-tested product image into a product defect detection model to obtain a product defect detection result output by the product defect detection model; wherein the product defect detection model is used for product defect detection on the to-be-tested product image according to the distance between the feature map of the to-be-tested product image and a target feature mapping center; and the product defect detection model is obtained by training sample product images in different positions and / or angles. The application is used to solve the defects in the prior art that product defect detection relies on manual work, which not only improves the labor cost, but also makes it difficult to guarantee the efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a product defect detection method, electronic equipment, storage medium, and program product. Background Art

[0002] With the development of automated production technology, the output of various industrial products is increasing year by year. Surface defect detection is particularly important in the production process, as it directly impacts product quality and downstream production processes. Currently, product appearance quality inspection is largely performed manually, which not only increases labor costs but also makes efficiency and accuracy difficult to guarantee. Therefore, there is an urgent need to automate the detection of product appearance defects. Summary of the Invention

[0003] The present invention provides a product defect detection method, electronic equipment, storage medium and program product, which are used to solve the problem that product defect detection in the prior art relies on manual labor, which not only increases labor costs but also makes it difficult to ensure efficiency and accuracy.

[0004] The present invention provides a product defect detection method, comprising:

[0005] Determine the product image to be tested;

[0006] Obtaining a product defect detection result, inputting the image of the product to be tested into a product defect detection model, and obtaining a product defect detection result output by the product defect detection model;

[0007] Among them, the product defect detection model is used to perform product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0008] According to a product defect detection method provided by the present invention, the product defect detection model includes a feature extraction layer, and the product defect detection model also includes an image matching layer and a defect area positioning layer;

[0009] The image matching layer is used to adjust the position and / or angle of the image of the product to be tested to the same position and / or angle as the first target image;

[0010] The defect area positioning layer is used to construct a first distance heat map according to the distance between the feature map of the product image to be tested and the center of the target feature map, and obtain the product defect detection result of the product image to be tested according to the first distance heat map.

[0011] According to a product defect detection method provided by the present invention, the image matching layer includes:

[0012] a sub-feature extraction layer, configured to obtain a first target image and perform feature extraction on the image of the product to be tested and the first target image through a preset feature extraction network, thereby obtaining a first feature map corresponding to the image of the product to be tested and a second feature map corresponding to the first target image;

[0013] A concatenation layer, configured to concatenate the first feature map and the second feature map in a channel dimension to obtain a third feature map;

[0014] The final matching layer is used to obtain geometric transformation parameters based on the third feature map, and match the image of the product to be tested to the same position and / or angle as the first target image according to the geometric transformation parameters to obtain a first matching image; the geometric transformation parameters include at least one of a horizontal translation amount, a vertical translation amount, and a rotation angle.

[0015] According to a product defect detection method provided by the present invention, the feature extraction layer is used to reduce the distance between the features of the normal image in the product image to be tested and the center of the target feature map, and to increase the distance between the features of the abnormal image in the product image to be tested and the center of the target feature map.

[0016] According to a product defect detection method provided by the present invention, before determining the image of the product to be tested, the method further includes the step of training the product defect detection model. The training of the product defect detection model includes:

[0017] Identify sample product images;

[0018] randomly rotating and / or translating the sample product image;

[0019] Adjusting the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image;

[0020] Extracting features from the sample product image using a target feature extraction network to obtain a target feature map; the target feature extraction network is used to reduce the distance between the features of normal images in the sample product image and the center of the target feature map, and to increase the distance between the features of abnormal images in the sample product image and the center of the target feature map;

[0021] constructing a second distance heat map based on the distance between each feature of the target feature map of the sample product image and the center of the target feature map;

[0022] A second loss function is constructed based on the second distance heat map, and the target feature extraction network is trained based on the second loss function.

[0023] According to a product defect detection method provided by the present invention, adjusting the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image includes:

[0024] Acquire a second target image, and perform feature extraction on the sample product image and the second target image using a preset feature extraction network to obtain a fourth feature map corresponding to the sample product image and a fifth feature map corresponding to the second target image, respectively;

[0025] splicing the fourth feature map and the fifth feature map in the channel dimension to obtain a sixth feature map;

[0026] Based on the sixth feature map, geometric transformation parameters are obtained, and the sample product image is matched to the same position and / or angle as the second target image according to the geometric transformation parameters to obtain a second matching image; the geometric transformation parameters include at least one of a horizontal translation amount, a vertical translation amount, and a rotation angle.

[0027] According to a product defect detection method provided by the present invention, the geometric transformation parameters are obtained by the following formula:

[0028]

[0029] Among them, I O For sample product images, I T is the second target image, N P is the preset feature extraction network, N P (I O ) is the fourth characteristic graph, N P (I T ) is the fifth feature map; Represents the process of concatenating the fourth feature map and the fifth feature map in the channel dimension, N R is the parameter regression network, N R Obtaining the geometric transformation parameters according to the spliced ​​sixth feature map;

[0030] The second matching image is obtained by the following formula:

[0031] I M =f geo (I O , T x , T y ,θ);

[0032] Among them, f geo Represents the process of geometric transformation, which transforms the sample product image I O The target in the image is matched to the second target image I T Same position and angle, IM is the second matching image obtained after matching.

[0033] According to a product defect detection method provided by the present invention, the training of the product defect detection model further includes:

[0034] calculating a first loss function based on the second matching image and the second target image;

[0035] The preset feature extraction network and the parameter regression network are trained according to the first loss function.

[0036] According to a product defect detection method provided by the present invention, the first loss function is obtained by the following formula:

[0037]

[0038] Among them, L reg is the first loss function, and n is the number of pixels in the sample product image.

[0039] According to a product defect detection method provided by the present invention, after adjusting the position and / or angle of the sample product image to the same position and / or angle as the target sample image, the method further includes:

[0040] An abnormal image is displayed in the target area in the sample product image, and the brightness or texture of the abnormal image is different from the brightness or texture of other areas in the sample product image except the target area.

[0041] According to a product defect detection method provided by the present invention, the second loss function is expressed by the following formula:

[0042]

[0043] Among them, L map (F M , F C ) represents the second loss function, F M is the target feature map of the sample product image, F C is the center of the target feature map; w and h represent the width and height of the feature map of the sample product image, respectively. s is a target feature map F with sample product images M binary images with the same resolution, Represents M s The coordinates are the values ​​at position (i, j), Represents the corresponding The region extracted from the abnormal image, and Represents the corresponding Regions extracted from normal images; Represents the value at the coordinate (i, j) in the second distance heat map; δ represents the upper limit of the feature distance of the abnormal image.

[0044] The present invention also provides a product defect detection device, comprising:

[0045] A determination module, used to determine the image of the product to be tested;

[0046] A detection result acquisition module is used to obtain product defect detection results, input the image of the product to be tested into the product defect detection model, and obtain the product defect detection results output by the product defect detection model;

[0047] Among them, the product defect detection model is used to perform product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0048] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described product defect detection methods are implemented.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described product defect detection methods.

[0050] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned product defect detection methods are implemented.

[0051] The product defect detection method, electronic device, storage medium, and program product provided by the present invention determine the image of the product to be tested; obtain the product defect detection result, input the product image to be tested into a product defect detection model, and obtain the product defect detection result output by the product defect detection model; wherein the product defect detection model is used to perform product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles. Since the normal area and defective area of ​​the product image to be tested are at different distances from the center of the target feature map, the present invention performs product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map to obtain the product defect detection result. Therefore, the present invention does not need to rely on manual labor, reduces labor costs, and improves the efficiency and accuracy of product defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is one of the flow charts of the product defect detection method provided by the present invention;

[0054] Figure 2 Schematic diagram of image matching performed by the present invention;

[0055] Figure 3 It is a schematic diagram of the product image to be tested, the first matching image, the defect area positioning result and the defect annotation image of the present invention;

[0056] Figure 4 This is the second flow chart of the product defect detection method provided by the present invention;

[0057] Figure 5 This is the third flow chart of the product defect detection method provided by the present invention;

[0058] Figure 6 This is the fourth flow chart of the product defect detection method provided by the present invention;

[0059] Figure 7 This is the fifth flow chart of the product defect detection method provided by the present invention;

[0060] Figure 8 This is the sixth flow chart of the product defect detection method provided by the present invention;

[0061] Figure 9 It is a schematic diagram of the synthesis of abnormal sample product images of the present invention;

[0062] Figure 10 It is a schematic diagram of feature extraction performed by the present invention;

[0063] Figure 11 It is a structural schematic diagram of the product defect detection device provided by the present invention;

[0064] Figure 12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0066] The following combination Figures 1-10 The product defect detection method of the present invention is described.

[0067] Please refer to Figure 1 The present invention provides a product defect detection method, comprising:

[0068] Step 200: Determine the image of the product to be tested;

[0069] The image of the product to be tested is obtained by the camera device of the electronic device, wherein the image of the product to be tested can be images of various industrial products, such as industrial cleaning products (such as toothbrushes), toys, tableware (such as bowls), etc.

[0070] Step 300: Obtain product defect detection results, input the image of the product to be tested into a product defect detection model, and obtain the product defect detection results output by the product defect detection model;

[0071] Among them, the product defect detection model is used to perform product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0072] Since there are differences in brightness, pattern or texture between the defective area and the normal area of ​​the image of the product to be tested, the distances between the defective area and the normal area of ​​the feature map of the product to be tested and the center of the target feature map are also different, so that product defect detection is performed on the image of the product to be tested based on the distance between the feature map of the product to be tested and the center of the target feature map. In addition, the product defect detection model of the present invention should make the difference between the distance between the defective area and the center of the target feature map of the feature map of the product to be tested, and the distance between the normal area and the center of the target feature map as large as possible. For example, for features extracted from the normal area, it is desirable to reduce the distance between the defective area and the center of the target feature map as much as possible, while for features extracted from the defective area, it is desirable to increase the distance between the defective area and the center of the target feature map as much as possible.

[0073] Furthermore, the product defect detection model is trained based on sample product images at different positions and / or angles. Since the original product image acquisition conditions are ideal, the position and / or angle of the scraper to be tested remain essentially the same. However, during typical production processes, the products to be tested often experience translation and / or rotational changes, and a model trained under ideal conditions would be unable to adapt to such complex environments. Therefore, the product defect detection model is trained based on sample product images at different positions and / or angles, simulating the differences in position and / or angle that may occur in actual testing environments. This improves the accuracy of the product defect detection model in detecting product defects and expands the application scenarios for product defect detection.

[0074] In some feasible embodiments, the product defect detection model includes a feature extraction layer, and the product defect detection model also includes an image matching layer and a defect area positioning layer.

[0075] The image matching layer is used to adjust the position and / or angle of the image of the product to be tested to the same position and / or angle as the first target image.

[0076] Traditional product anomaly detection methods often fail to account for variations in the position and / or angle of the product under test. Instead, they often use the same model to describe these diverse samples, which can affect the localization of subtle defects. In this invention, image matching is first performed. The image matching layer analyzes the differences between the original and target images and attempts to adjust the product under test to the same position and / or angle as the target image.

[0077] Furthermore, the image matching layer includes:

[0078] The sub-feature extraction layer is used to obtain a first target image and perform feature extraction on the image of the product to be tested and the first target image through a preset feature extraction network to respectively obtain a first feature map corresponding to the image of the product to be tested and a second feature map corresponding to the first target image.

[0079] For details, please refer to Figure 2 , the image of the product to be tested I O and the first target image I randomly selected from the training set T , first pass through the preset feature extraction network N P Perform feature extraction and obtain the corresponding first feature map F O and the second feature map F T .

[0080] A concatenation layer is configured to concatenate the first feature map and the second feature map in a channel dimension to obtain a third feature map.

[0081] The final matching layer is used to obtain geometric transformation parameters based on the third feature map, and match the image of the product to be tested to the same position and / or angle as the first target image according to the geometric transformation parameters to obtain a first matching image.

[0082] According to the spliced ​​third feature map, the geometric transformation related parameters can be obtained: for example, the horizontal translation T x , vertical translation T y And the rotation angle θ. Specifically:

[0083]

[0084] Among them, I O is the image of the product to be tested, I T is the first target image randomly selected from the training set, N P is the preset feature extraction network, N P (I O ) represents the first feature map, N P (I T ) represents the second feature map; Represents the process of concatenating two feature maps in the channel dimension, N R is the parameter regression network, N R The parameters of three geometric transformations are obtained based on the spliced ​​feature maps.

[0085] It is worth mentioning that, for the case where the image of the product to be tested changes position compared with the first target image, the geometric transformation parameters obtained according to the spliced ​​third feature map are the horizontal translation Tx and the vertical translation Ty; for the case where the image of the product to be tested changes rotationally compared with the first target image, the geometric transformation parameter obtained according to the spliced ​​third feature map is the rotation angle θ; for the case where the image of the product to be tested changes position and rotationally at the same time compared with the first target image, the geometric transformation parameters obtained according to the spliced ​​third feature map are the horizontal translation Tx, the vertical translation Ty and the rotation angle θ.

[0086] Please refer to Figure 2 The electronic device rotates and translates the image of the product to be tested I O The target in the image is matched to the same position and angle as the first target image IT, and the first matching image I is obtained. M Specifically,

[0087] I M =f geo (I O , T x , T y ,θ); Formula (2)

[0088] Among them, f geoRepresents the process of geometric transformation, which transforms the image of the product to be tested I O The target in the image is matched to the first target image I T Same position and angle, I M is the first matching image obtained after matching.

[0089] The image matching layer is used to eliminate the differences in position and / or angle of the product to be tested in the product image in advance, so as to facilitate the training of the subsequent feature extraction layer.

[0090] After the image matching layer, this embodiment first specifies a target feature mapping center. In the process of feature extraction using the feature extraction layer, the distance from the features of the normal area in the image of the product to be tested to the target feature mapping center is reduced as much as possible, while the distance from the features of the abnormal area in the image of the product to be tested to the target feature mapping center is increased as much as possible.

[0091] This embodiment uses the feature map as the mapping center, and assigns a target feature map center point to each local image region. For features extracted from normal regions, the distance between them and the corresponding target feature map center is minimized, while for features extracted from abnormal image regions, the distance between them and the corresponding target feature map center is maximized.

[0092] The defect area positioning layer is used to construct a first distance heat map according to the distance between the feature map of the product image to be tested and the center of the target feature map, and obtain the product defect detection result of the product image to be tested according to the first distance heat map.

[0093] The electronic device extracts features from each image area of ​​the product to be tested and compares them with the target feature mapping center F C The cosine distance is calculated for the corresponding feature vector in [ ] and all distance values ​​are counted to construct the first distance heatmap. Because the resolution of the feature map and the calculated first distance heatmap is smaller than the image of the product to be tested, the defect area can be located by upsampling.

[0094] Among them, the first distance heat map is expressed by the following formula:

[0095]

[0096] Among them, F M is the feature map of the product image to be tested, F C is the target feature map center, Indicates the cosine distance between the feature map of the product image to be tested and the center of the target feature map at coordinate (i, j).

[0097] In a typical embodiment, the defect area positioning effect is as follows: Figure 3 As shown, from left to right they represent the image of the product to be tested, the first matching image, the defect area positioning result and the defect annotation map. The brighter the area in the positioning result, the more likely it is a defect area (or abnormal image area). The method proposed in the present invention has been tested in a public data set, and its average positioning index area under the curve (Area Under the Curve) has reached 0.98, and it only takes 27ms to complete the positioning of the defect area in a 224×224 pixel image. Therefore, the present invention performs product defect detection on the image of the product to be tested by the distance between the feature map of the image of the product to be tested and the center of the target feature map, and obtains the product defect detection result. The present invention does not need to rely on manual labor, reduces labor costs, and improves the efficiency and accuracy of product defect detection.

[0098] By determining the image of the product to be tested; and obtaining the product defect detection result, the product image to be tested is input into the product defect detection model, and the product defect detection result output by the product defect detection model is obtained; wherein the product defect detection model is used to perform product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map; the product defect detection model is obtained by training sample product images at different positions and / or angles. Since the normal area and the defective area of ​​the product image to be tested are at different distances from the center of the target feature map, the present invention performs product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map to obtain the product defect detection result. Therefore, the present invention does not need to rely on manual labor, reduces labor costs, and improves the efficiency and accuracy of product defect detection.

[0099] In some other embodiments, please refer to Figure 4 Before step 200, determining the image of the product to be tested, the method further includes:

[0100] Step 100: training the product defect detection model. Figure 5 , the training of the product defect detection model includes:

[0101] Step 110: Determine a sample product image;

[0102] Similarly, sample product images are obtained through the camera device of the electronic device, where the sample product images can be images of various industrial products, such as industrial cleaning products (such as toothbrushes), toys, tableware (such as bowls), cloth, medicine (such as capsules), connectors (such as screws), and other industrial products.

[0103] It's worth noting that existing methods for detecting appearance defects often require the collection and manual labeling of a large number of defect samples before model training and parameter adjustment can proceed. This significantly increases the R&D cost and cycle time of the detection system, and also makes existing methods difficult to apply to scenarios where production lines are frequently replaced. However, the sample product images of this embodiment can be trained using various normal, defect-free product images, without requiring any labeled defect sample images. Only normal sample product images are required for training, making the training method highly practical and versatile.

[0104] Step 120: randomly rotating and / or translating the sample product image;

[0105] The electronic device randomly rotates and / or translates the sample product image. This embodiment pre-processes the data by randomly rotating and / or translating the sample product image to simulate the differences in position and angle that sample products may appear in an actual inspection environment. All sample product images are randomly rotated and translated.

[0106] In some feasible embodiments, optionally, the range of rotation is plus or minus 10 degrees, and the range of translation is 0-32 pixels in the upward, downward, left or right direction. By randomly rotating and / or translating the sample product image, the diversity of the sample is increased to better simulate the different positions and / or angles of the sample products in reality.

[0107] Step 130: Adjust the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image;

[0108] The electronic device first performs image matching training on a sample product by analyzing the differences between the sample product image and the target image and attempting to adjust the sample product image to the same position and / or angle as the target image.

[0109] For details, please refer to Figure 6 Step 130, adjusting the position and / or angle of the sample product image to the same position and / or angle as the target sample image, includes:

[0110] Step 131: Acquire a second target image, and perform feature extraction on the sample product image and the second target image through a preset feature extraction network to obtain a fourth feature map corresponding to the sample product image and a fifth feature map corresponding to the second target image, respectively.

[0111] Specifically, the sample product image and the second target image randomly selected from the training set are first subjected to feature extraction by a preset feature extraction network to obtain the corresponding fourth feature map and fifth feature map respectively.

[0112] Step 132: Concatenate the fourth feature map and the fifth feature map in the channel dimension to obtain a sixth feature map.

[0113] Step 133: Acquire geometric transformation parameters based on the sixth feature map, and match the sample product image to the same position and / or angle as the second target image according to the geometric transformation parameters to obtain a second matching image; the geometric transformation parameters include at least one of a horizontal translation amount, a vertical translation amount, and a rotation angle.

[0114] According to the spliced ​​sixth feature map, the geometric transformation related parameters are obtained: for example, the horizontal translation T x , vertical translation T y And the rotation angle θ. Specifically: See formula (1)

[0115] At this time, I in formula (1) O For sample product images, I T is the second target image randomly selected from the training set, N P is the preset feature extraction network, N P (I O ) represents the fourth feature map, N P (I T ) represents the fifth feature map; Represents the process of concatenating two feature maps in the channel dimension, N R is the parameter regression network, N R The parameters of three geometric transformations are obtained based on the spliced ​​feature maps.

[0116] It is worth mentioning that, for the case where the position of the sample product image changes compared with the second target image, the geometric transformation parameters obtained according to the sixth feature map after splicing are the horizontal translation Tx and the vertical translation Ty; for the case where the sample product image changes in rotation compared with the second target image, the geometric transformation parameter obtained according to the sixth feature map after splicing is the rotation angle θ; for the case where the sample product image changes in position and rotation at the same time compared with the second target image, the geometric transformation parameters obtained according to the sixth feature map after splicing are the horizontal translation Tx, the vertical translation Ty and the rotation angle θ.

[0117] The electronic device matches the target in the sample product image to the same position and angle as the second target image by rotation and / or translation, thereby obtaining a second matching image. For details, please refer to formula (2). geo Represents the process of geometric transformation, which matches the target in the sample product image to the same position and angle as the second target image by rotation and translation, I M is the second matching image obtained after matching.

[0118] By performing image matching between the sample product image and the second target image, differences in position and / or angle of the sample product in the sample product image can be eliminated in advance, facilitating subsequent feature extraction training.

[0119] It is worth mentioning that please refer to Figure 7 Step 100, adjusting the position and / or angle of the sample product image to the same position and / or angle as the target sample image, further includes:

[0120] Step 134: Calculate a first loss function based on the second matching image and the second target image;

[0121] Step 135: Train the preset feature extraction network and the parameter regression network according to the first loss function.

[0122] The first loss function formula is as follows:

[0123] Among them, L reg is the first loss function, n is the number of pixels in the second matching image, and formula (4) is calculated by M With the second target image I T The difference between the two is used to train the preset feature extraction network N P and parameter regression network N R .

[0124] In step 130 of this embodiment, the target in the sample product image may be adjusted to correspond to the second target image I T The same position and / or angle greatly reduces the difficulty of subsequent feature extraction and anomaly detection.

[0125] In some other embodiments, please refer to Figure 8 After step 130, adjusting the position and / or angle of the image of the product to be tested to the same position and / or angle as the target sample image, the method further includes:

[0126] Step 140: Display an abnormal image in the target area of ​​the sample product image, wherein the brightness or texture of the abnormal image is different from the brightness or texture of other areas of the sample product image except the target area.

[0127] Existing anomaly detection methods are often trained only on normal samples, and only learn relevant knowledge about normal samples, which limits the detection accuracy of actual defective samples. This embodiment automatically generates a variety of anomaly samples (or defective samples) to improve the training effect.

[0128] The electronic device displays an abnormal image in the target area of ​​the sample product image, and the brightness or texture of the abnormal image is different from the brightness or texture of other areas of the sample product image except the target area. Figure 9 It represents the synthesis diagram of abnormal sample product images. In the normal sample product image I O The second matching image I obtained after matching M In the example, the abnormal image M is superimposed to finally form an abnormal sample product image I with abnormal patterns in the local area. M+ These abnormal sample product images I M+ To a certain extent, it simulates real defect images, which helps to improve the discrimination ability and detection accuracy of product defect detection models.

[0129] The abnormal image can be a shape (such as a circle, square, rectangle, triangle or irregular shape, etc.), or a multi-segment broken line with random parameters such as length, position, angle, and width. The image in the area covered by the broken line will randomly become brighter, darker, or directly replaced with an image from another dataset.

[0130] It is necessary to input abnormal sample product image I M+ Perform feature extraction and detect product defects by analyzing the extracted features.

[0131] Step 150: extracting features from the sample product image using a target feature extraction network to obtain a target feature map; the target feature extraction network is configured to reduce the distance between features of normal images in the sample product image and the center of the target feature map, and to increase the distance between features of abnormal images in the sample product image and the center of the target feature map;

[0132] The present invention adopts the feature map as the target feature mapping center, and specifies a target feature mapping center point for each local area of ​​the sample product image. Figure 10 Shows a schematic diagram of feature extraction in the present invention, wherein I M+ is the input abnormal sample product image, N M is the target feature extraction network, F M is the target feature map of the sample product image, F C The target feature map center is obtained by extracting features from the sample product image using a target feature extraction network. For features extracted from normal areas of the sample product image, the distance between the features and the target feature map center is minimized, while for features extracted from abnormal areas of the sample product image, the distance between the features and the target feature map center is maximized.

[0133] Step 160: construct a second distance heat map based on the distance between each feature of the target feature map of the sample product image and the center of the target feature map;

[0134] Among them, the second distance heat map is expressed by the following formula:

[0135]

[0136] Among them, F M is the target feature map of the sample product image, F C is the target feature map center, Represents the value at the coordinate (i, j) in the second distance heat map.

[0137] Step 170: construct a second loss function based on the second distance heat map, and train the target feature extraction network based on the second loss function.

[0138] Specifically, the second loss function is expressed by the following formula:

[0139]

[0140]

[0141] Among them, L map (F M , F C ) represents the second loss function, F M is the target feature map of the sample product image, F C is the center of the target feature map; w and h represent the width and height of the feature map of the sample product image, respectively. s is a target feature map F with sample product images M binary images with the same resolution, Represents M s The coordinates are the values ​​at position (i, j), Represents the corresponding The region extracted from the abnormal image, and Represents the corresponding Regions extracted from normal images; Represents the value at the coordinate (i, j) in the second distance heat map; δ represents the upper limit of the feature distance of the abnormal image.

[0142] It should be noted that, among them, F M =N M (I M+ ), F M Represents a sample product image I O The target feature map, N M express Figure 10Target feature extraction network N M , I M+ Indicates that in the normal sample product image I O The second matching image I obtained after matching M In the example, the abnormal image M is superimposed to finally form an abnormal sample product image with abnormal patterns in the local area.

[0143] The first half of formula (6) minimizes the distance between the features of the normal image and the center point of the corresponding target feature map, while the second half of formula (6) maximizes the distance between the features of the abnormal image and the center point of the target feature map.

[0144] It is worth mentioning that for formula (6), this embodiment sets an upper limit value δ of the abnormal region feature distance. When the feature of the abnormal region is far enough from the center point of the target feature map, that is, When the max function is used, the second half of formula (6) is no longer involved in the calculation, so as to avoid step 170 focusing too much on the abnormal sample product image I in step 140. M+ Optimization leads to overfitting.

[0145] The electronic device extracts features from each image area in the sample product image and compares them with the target feature mapping center F C The cosine distance is calculated for the corresponding feature vector in [ ] and all distance values ​​are counted to construct a second distance heatmap. Because the resolution of the target feature map and the calculated second distance heatmap of the sample product image is smaller than that of the sample product image, upsampling is used to locate the defect area in the sample product image.

[0146] The product defect detection device provided by the present invention is described below. The product defect detection device described below and the product defect detection method described above can be referenced to each other.

[0147] Please refer to Figure 11 The present invention also provides a product defect detection device, comprising:

[0148] A first determining module 201 is used to determine an image of a product to be tested;

[0149] The first detection result acquisition module 202 is used to obtain a product defect detection result by inputting the image of the product to be tested into a product defect detection model and obtaining a product defect detection result output by the product defect detection model;

[0150] Among them, the product defect detection model is used to perform product defect detection on the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0151] Based on the above embodiments, as an optional embodiment, the product defect detection model includes a first feature extraction module, and the product defect detection model also includes a first image matching module and a first defect area positioning module;

[0152] The first image matching module is used to adjust the position and / or angle of the image of the product to be tested to the same position and / or angle as the first target image;

[0153] The first defect area positioning module is used to construct a first distance heat map according to the distance between the feature map of the product image to be tested and the center of the target feature map, and obtain a product defect detection result of the product image to be tested according to the first distance heat map.

[0154] Based on the above embodiments, as an optional embodiment, the first image matching module includes:

[0155] a first sub-feature extraction module, configured to obtain a first target image and perform feature extraction on the image of the product to be tested and the first target image using a preset feature extraction network, to respectively obtain a first feature map corresponding to the image of the product to be tested and a second feature map corresponding to the first target image;

[0156] A first splicing module, configured to splice the first feature map and the second feature map in a channel dimension to obtain a third feature map;

[0157] A first final matching module is configured to obtain geometric transformation parameters based on the third feature map, and match the image of the product to be tested to the same position and / or angle as the first target image according to the geometric transformation parameters to obtain a first matching image; the geometric transformation parameters include at least one of a horizontal translation amount, a vertical translation amount, and a rotation angle.

[0158] Based on the above embodiments, as an optional embodiment, the first feature extraction module is used to reduce the distance from the features of the normal image in the product image to be tested to the center of the target feature mapping, and increase the distance from the features of the abnormal image in the product image to be tested to the center of the target feature mapping.

[0159] Based on the above embodiments, as an optional embodiment, the product defect detection device further includes a model training module, which includes:

[0160] A second determining module is used to determine a sample product image;

[0161] a geometric transformation module, configured to randomly rotate and / or translate the sample product image;

[0162] A second image matching module is used to adjust the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image;

[0163] a second feature extraction module, configured to extract features from the sample product image using a target feature extraction network to obtain a target feature map; the target feature extraction network is configured to reduce the distance between features of normal images in the sample product image and the center of the target feature map, and to increase the distance between features of abnormal images in the sample product image and the center of the target feature map;

[0164] A second distance heat map construction module is used to construct a second distance heat map based on the distance between each feature of the target feature map of the sample product image and the center of the target feature map;

[0165] A training module is used to construct a second loss function based on the second distance heat map, and train the target feature extraction network based on the second loss function.

[0166] Based on the above embodiments, as an optional embodiment, the second image matching module includes:

[0167] a second sub-feature extraction module, configured to obtain a second target image, and perform feature extraction on the sample product image and the second target image using a preset feature extraction network, to obtain a fourth feature map corresponding to the sample product image and a fifth feature map corresponding to the second target image, respectively;

[0168] A second splicing module, configured to splice the fourth feature map and the fifth feature map in a channel dimension to obtain a sixth feature map;

[0169] A second final matching module is used to obtain geometric transformation parameters based on the sixth feature map, and match the sample product image to the same position and / or angle as the second target image according to the geometric transformation parameters to obtain a second matching image; the geometric transformation parameters include at least one of horizontal translation, vertical translation and rotation angle.

[0170] Based on the above embodiments, as an optional embodiment, the geometric transformation parameters are obtained by the following formula:

[0171]

[0172] Among them, I O For sample product images, IT is the second target image, N P is the preset feature extraction network, N P (I O ) is the fourth characteristic graph, N P (I T ) is the fifth feature map; Represents the process of concatenating the fourth feature map and the fifth feature map in the channel dimension, N R is the parameter regression network, N R Obtaining the geometric transformation parameters according to the spliced ​​sixth feature map;

[0173] The second matching image is obtained by the following formula:

[0174] I M =f geo (I O , T x , T y ,θ);

[0175] Among them, f geo Represents the process of geometric transformation, which transforms the sample product image I O The target in the image is matched to the second target image I T Same position and angle, I M is the second matching image obtained after matching.

[0176] Based on the above embodiments, as an optional embodiment, the model training module further includes:

[0177] A first loss function calculation module, configured to calculate a first loss function based on the second matching image and the second target image;

[0178] A first loss function training module is used to train the preset feature extraction network and the parameter regression network according to the first loss function.

[0179] Based on the above embodiments, as an optional embodiment, the first loss function is obtained by the following formula:

[0180]

[0181] Among them, L reg is the first loss function, and n is the number of pixels in the sample product image.

[0182] Based on the above embodiments, as an optional embodiment, the product defect detection device further includes:

[0183] The abnormal sample construction module is used to display an abnormal image in the target area of ​​the sample product image, wherein the brightness or texture of the abnormal image is different from the brightness or texture of other areas of the sample product image except the target area.

[0184] Based on the above embodiments, as an optional embodiment, the second loss function is obtained by the following formula:

[0185]

[0186] Among them, L map (F M , F C ) represents the second loss function, F M is the target feature map of the sample product image, F C is the center of the target feature map; w and h represent the width and height of the feature map of the sample product image, respectively. s is a target feature map F with sample product images M binary images with the same resolution, Represents M s The coordinates are the values ​​at position (i, j), Represents the corresponding The region extracted from the abnormal image, and Represents the corresponding Regions extracted from normal images; Represents the value at the position with coordinates (i, j) in the second distance heat map; δ represents the upper limit of the characteristic distance of the abnormal image.

[0187] Figure 12 An example of a physical structure diagram of an electronic device is shown below. Figure 12 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a product defect detection method, which includes: determining an image of a product to be tested; obtaining a product defect detection result, inputting the image of the product to be tested into a product defect detection model, and obtaining a product defect detection result output by the product defect detection model; wherein the product defect detection model is used to perform product defect detection on the image of the product to be tested based on the distance between the feature map of the image of the product to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0188] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0189] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the product defect detection method provided by the above methods, which includes: determining the image of the product to be tested; obtaining the product defect detection result, inputting the image of the product to be tested into a product defect detection model, and obtaining the product defect detection result output by the product defect detection model; wherein the product defect detection model is used to perform product defect detection on the image of the product to be tested based on the distance between the feature map of the image of the product to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0190] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the product defect detection method provided by the above-mentioned methods, the method comprising: determining an image of a product to be tested; obtaining a product defect detection result, inputting the image of the product to be tested into a product defect detection model, and obtaining a product defect detection result output by the product defect detection model; wherein the product defect detection model is used to perform product defect detection on the image of the product to be tested based on the distance between the feature map of the image of the product to be tested and the center of the target feature map; the product defect detection model is trained based on sample product images at different positions and / or angles.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0192] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A product defect detection method, characterized in that: include: Determine the product image to be tested; Obtaining a product defect detection result, inputting the image of the product to be tested into a product defect detection model, and obtaining a product defect detection result output by the product defect detection model; The product defect detection model is used to detect product defects in the product image to be tested based on the distance between the feature map of the product image to be tested and the center of the target feature map. The product defect detection model is trained based on sample product images at different positions and / or angles. The product defect detection model includes a feature extraction layer, an image matching layer, and a defect area location layer. The target feature map center is a feature map specified in each local image area of ​​the product image to be tested. The image matching layer is used to adjust the position and / or angle of the product image to be tested to be the same as the position and / or angle of the first target image. The feature extraction layer is used to reduce the distance between the features of the normal area of ​​the product image to be tested and the center of the target feature map, and increase the distance between the features of the abnormal area of ​​the product image to be tested and the center of the target feature map. The first target image is randomly selected from a training set, and the training set is sample product images at different positions and / or angles. The defect area positioning layer is used to construct a first distance heat map according to the distance between the feature map of the product image to be tested and the center of the target feature map, and obtain a product defect detection result of the product image to be tested according to the first distance heat map; Before determining the image of the product to be tested, the method further includes a step of training the product defect detection model, wherein the training of the product defect detection model includes: Determine a sample product image, where the sample product image is a normal product image without defects; randomly rotating and / or translating the sample product image; Adjusting the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image; Extracting features from the sample product image using a target feature extraction network to obtain a target feature map; the target feature extraction network is used to reduce the distance between features in a normal area of ​​the sample product image and the center of the target feature map, and to increase the distance between features in an abnormal area of ​​the sample product image and the center of the target feature map; constructing a second distance heat map based on the distance between each feature of the target feature map of the sample product image and the center of the target feature map; A second loss function is constructed based on the second distance heat map, and the target feature extraction network is trained based on the second loss function.

2. The product defect detection method according to claim 1, characterized in that: The image matching layer includes: a sub-feature extraction layer, configured to obtain a first target image and perform feature extraction on the image of the product to be tested and the first target image through a preset feature extraction network, thereby obtaining a first feature map corresponding to the image of the product to be tested and a second feature map corresponding to the first target image; A concatenation layer, configured to concatenate the first feature map and the second feature map in a channel dimension to obtain a third feature map; The final matching layer is used to obtain geometric transformation parameters based on the third feature map, and match the image of the product to be tested to the same position and / or angle as the first target image according to the geometric transformation parameters to obtain a first matching image; the geometric transformation parameters include at least one of a horizontal translation amount, a vertical translation amount, and a rotation angle.

3. The product defect detection method according to claim 2, characterized in that: The adjusting the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image includes: Acquire a second target image, and perform feature extraction on the sample product image and the second target image using a preset feature extraction network to respectively acquire a fourth feature map corresponding to the sample product image and a fifth feature map corresponding to the second target image, wherein the second target image is randomly selected from the training set, which is the sample product images at different positions and / or angles; splicing the fourth feature map and the fifth feature map in the channel dimension to obtain a sixth feature map; Based on the sixth feature map, geometric transformation parameters are obtained, and the sample product image is matched to the same position and / or angle as the second target image according to the geometric transformation parameters to obtain a second matching image; the geometric transformation parameters include at least one of a horizontal translation amount, a vertical translation amount, and a rotation angle.

4. The product defect detection method according to claim 3, characterized in that: The geometric transformation parameters are obtained by the following formula: ； in, I O For sample product images, I T is the second target image, N P is the preset feature extraction network, N P (I O ) is the fourth characteristic graph, N P (I T ) is the fifth feature map; Represents the process of concatenating the fourth feature map and the fifth feature map in the channel dimension. N R is a parameter regression network, N R Obtaining the geometric transformation parameters according to the spliced ​​sixth feature map; The second matching image is obtained by the following formula: ； in, f geo Represents the process of geometric transformation, which transforms the sample product image into I O The target in the image is matched to the second target image I T Same position and angle, I M is the second matching image obtained after matching.

5. The product defect detection method according to claim 4, characterized in that: The training of the product defect detection model further includes: calculating a first loss function based on the second matching image and the second target image; The preset feature extraction network and the parameter regression network are trained according to the first loss function.

6. The product defect detection method according to claim 5, characterized in that: The first loss function is obtained by the following formula: ； in, L reg is the first loss function, n is the number of pixels in the sample product image.

7. The product defect detection method according to claim 1, characterized in that: After adjusting the position and / or angle of the sample product image to be the same as the position and / or angle of the target sample image, the method further includes: An abnormal image is displayed in the target area in the sample product image, and the brightness or texture of the abnormal image is different from the brightness or texture of other areas in the sample product image except the target area.

8. The product defect detection method according to claim 7, characterized in that: Also includes: The second loss function is expressed by the following formula: ； Among them, L map (F M， F C ) represents the second loss function, F M is the target feature map of the sample product image, F C is the center of the target feature map; w and h represent the width and height of the feature map of the sample product image, respectively. is a target feature map F with sample product images M binary images with the same resolution, Represents The coordinates are the values ​​at position (i, j), =1 represents the corresponding The region extracted from the abnormal image, and =0 represents the corresponding Regions extracted from normal images; Represents the value at the coordinate (i, j) in the second distance heat map; δ represents the upper limit of the feature distance of the abnormal image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the product defect detection method according to any one of claims 1 to 8 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the product defect detection method according to any one of claims 1 to 8 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the product defect detection method according to any one of claims 1 to 8 are implemented.

Images