A method, apparatus, equipment and storage medium for detecting suture defects

Abstract

This application discloses a method, apparatus, device, and storage medium for detecting suture defects, relating to the field of artificial intelligence technology. The method includes: acquiring an initial image of a sewn product to be inspected; performing keypoint detection on the initial image using a keypoint detection model; extracting a Region of Interest (ROI) image from the initial image based on the detected keypoints; inputting the ROI image into a suture defect detection model to obtain the detection results output by each defect detection sub-model within the suture defect detection model; and determining whether the sewn product to be inspected has suture defects based on the detection results; wherein the defect detection sub-model is a detection sub-model constructed according to the type of suture defect. By locating the sewn product using keypoints, the positioning accuracy and generalization performance are improved, and different defect types are analyzed from multiple perspectives, thereby enhancing the robustness of suture defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a sewing defect detection method, device, equipment and storage medium. BACKGROUND

[0002] At present, sewing products such as woven bags are often used in production and life, which mainly serve to package or store objects, greatly facilitating people's life and being widely used in the packaging of industrial and agricultural products. In the industrial production line, the packaging equipment needs to supervise the packaging result when packaging woven bags in batches, but due to some deficiencies in the packaging equipment process itself, it is easy to cause defects such as wireless, broken line, line connection and opening of the woven bag. At present, the packaging equipment can only stop processing for part of the defects and cannot achieve high-precision detection, and the missed defect samples will cause great loss of goods, in addition, additional manpower is needed for further screening, wasting manpower and resources. Moreover, due to the complexity of the type of sewing products after sewing, in addition to the influence of external environment such as light and dust, non-rigid sewing products are extremely prone to deformation and produce various wrinkle textures, therefore, how to improve the robustness of sewing defect detection of sewing products is a problem to be solved at present. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a sewing defect detection method, device, equipment and medium, which can improve the robustness of sewing defect detection of sewing products. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a sewing defect detection method, comprising:

[0005] obtaining an initial image of a sewing product to be detected, and detecting key points in the initial image by using a key point detection model;

[0006] extracting an ROI region image from the initial image according to the detected key points;

[0007] inputting the ROI region image into a sewing defect detection model to obtain detection results output by each defect detection sub-model in the sewing defect detection model, and judging whether the sewing product to be detected has a sewing defect based on the detection results; the defect detection sub-model is a detection sub-model constructed according to a sewing defect type.

[0008] Optionally, the key point detection model construction process comprises:

[0009] obtaining a plurality of historical images containing sewing products, adding corresponding key points on the historical images according to a sewing product key point marking rule and a sewing thread key point marking rule to obtain a key point marked image; the key points include sewing product key points and sewing thread key points;

[0010] training the key point detection model by using the key point marked image.

[0011] Optionally, after the key point detection model is used to detect the key points of the initial image, the method further comprises:

[0012] determining whether the current image is empty according to the distance between each key point of the sewn product and the origin coordinate in combination with a first preset threshold value;

[0013] determining whether the current image is a partial snapshot according to the horizontal distance between the upper left key point of the sewn product and the upper right key point of the sewn product in combination with a second preset threshold value;

[0014] determining whether the knitted product has a wire defect according to the distance between each key point of the sewing line and the origin coordinate in combination with a third preset threshold value;

[0015] determining whether the knitted product has a line overlapping defect or a line break defect according to the horizontal distance between the left boundary key point of the sewing line and the right boundary key point of the sewing line in combination with a fourth preset threshold value;

[0016] determining the inclination degree of the sewn product according to the inclination angle between the left boundary key point of the sewing line and the right boundary key point of the sewing line and a preset angle range;

[0017] if the inclination degree is an adjustable degree, performing image correction on the initial image;

[0018] if the inclination degree is a serious inclination, determining that the initial image has a serious inclination defect.

[0019] Optionally, the defect detection sub-model construction process comprises:

[0020] obtaining a normal image of the sewn product under normal sewing and defect images under different sewing defect types; the sewing defect types include any two or more of the following: opening defect, wire defect, line break defect and line overlapping defect;

[0021] constructing a corresponding defect detection initial sub-model for each sewing defect type;

[0022] generating training data corresponding to different sewing defect types based on the normal image and the defect image;

[0023] performing model training on the defect detection initial sub-model of the corresponding type by using the training data corresponding to different sewing defect types, to obtain the defect detection sub-model corresponding to different sewing defect types.

[0024] Optionally, the model training of the initial sub-model for defect detection of the corresponding type of defect by using the training data corresponding to different types of stitching defects comprises:

[0025] If the type of stitching defect is a wireless defect, the suture interval segment between each two adjacent suture key points is determined according to the suture key points on the ROI region image corresponding to the normal image.

[0026] A non-suture pixel region of the same size adjacent to the suture interval segment is selected, and the non-suture pixel region is used to replace the suture interval segment, until all the suture interval segments are replaced to obtain a simulated wireless defect image.

[0027] The normal image, the simulated wireless defect image and the wireless defect image are used as the training data corresponding to the wireless defect.

[0028] Optionally, the model training of the initial sub-model for defect detection of the corresponding type of defect by using the training data corresponding to different types of stitching defects comprises:

[0029] If the type of stitching defect is a broken line defect, the suture interval segment between each two adjacent suture key points is determined according to the suture key points on the ROI region image corresponding to the normal image, and part of the suture interval segments are randomly selected as target suture interval segments from all the suture interval segments.

[0030] A non-suture pixel region of the same size adjacent to the target suture interval segment is selected, and the non-suture pixel region is used to replace the target suture interval segment, until all the target suture interval segments are replaced to obtain a simulated broken line defect image.

[0031] The normal image, the simulated broken line defect image and the broken line defect image are used as the training data corresponding to the broken line defect.

[0032] Optionally, the generation of the training data corresponding to different types of stitching defects based on the normal image and the defect image comprises:

[0033] According to the watermarks in the watermark information library, watermarks are randomly added to the ROI region image of the normal image and the ROI region image of the defect image.

[0034] The training data corresponding to different types of stitching defects is generated based on the normal image and the defect image to which the watermarks are randomly added.

[0035] In a second aspect, the application discloses a stitching defect detection device, comprising:

[0036] The key point detection module is configured to acquire an initial image of a to-be-detected sewing product, and perform key point detection on the initial image by using a key point detection model.

[0037] The region extraction module is configured to extract an ROI region image from the initial image according to the detected key points.

[0038] The defect detection module is configured to input the ROI region image into a sewing defect detection model, obtain detection results output by each defect detection sub-model in the sewing defect detection model, and determine whether the to-be-detected sewing product has a sewing defect based on the detection results. The defect detection sub-model is a detection sub-model constructed according to a sewing defect type.

[0039] In a third aspect, the present application discloses an electronic device, comprising:

[0040] The memory is configured to save a computer program.

[0041] The processor is configured to execute the computer program to implement the above-mentioned sewing defect detection method.

[0042] In a fourth aspect, the present application discloses a computer readable storage medium configured to store a computer program, wherein the computer program is executed by a processor to implement the above-mentioned sewing defect detection method.

[0043] In the present application, an initial image of a to-be-detected sewing product is acquired, and key point detection is performed on the initial image by using a key point detection model. An ROI region image is extracted from the initial image according to the detected key points. The ROI region image is input into a sewing defect detection model, detection results output by each defect detection sub-model in the sewing defect detection model are obtained, and it is determined whether the to-be-detected sewing product has a sewing defect based on the detection results. The defect detection sub-model is a detection sub-model constructed according to a sewing defect type. It can be seen that the sewing defect detection of the sewing product integrates the key point detection of the sewing thread and the defect detection sub-models of multiple defect types as a whole. The positioning accuracy and the generalization performance are improved by positioning the sewing product through the key points, and the robustness of the sewing defect detection of the sewing product is improved by analyzing different defect types from multiple perspectives. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0045] Figure 1A stitching defect detection method flow chart is provided for the present application.

[0046] Figure 2 A specific sewing product initial image schematic diagram is provided for the present application.

[0047] Figure 3 A specific normal image and defect image schematic diagram is provided for the present application.

[0048] Figure 4 A specific woven bag stitching defect detection flow chart is provided for the present application.

[0049] Figure 5 A specific stitching defect detection method flow chart is provided for the present application.

[0050] Figure 6 A stitching defect detection device structure schematic diagram is provided for the present application.

[0051] Figure 7 An electronic device structure diagram is provided for the present application. DETAILED DESCRIPTION

[0052] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] In the prior art, the packaging equipment can only handle part of the defects for shutdown and cannot achieve high-precision detection for the above-mentioned defects. The missed defect samples will cause great loss of goods, and additional manpower is needed for further screening, wasting manpower and resources. To overcome the above technical problems, the present application provides a stitching defect detection method, which can improve the positioning accuracy and generalization performance, and improve the robustness of the sewing product stitching defect detection.

[0054] The embodiments of the present application disclose a stitching defect detection method, referring to Figure 1 The method can include the following steps:

[0055] Step S11: acquiring an initial image of a sewing product to be detected, and detecting key points of the initial image by using a key point detection model.

[0056] In this embodiment, first, an initial image of a to-be-detected sewing product is acquired, for example, a photo of a sewing product taken by a camera on a sewing product production line, and the sewing product can be a woven bag. Then, a key point detection model is used to detect key points on the initial image, and the key points include sewing product key points and sewing line key points. It can be understood that the sewing line key point detection of the woven bag is mainly used to accurately locate the positions of the woven bag and the sewing line.

[0057] In this embodiment, the key point detection model construction process can include: acquiring a plurality of historical images containing sewing products, adding corresponding key points on the historical images according to sewing product key point marking rules and sewing line key point marking rules to obtain key point marked images; the key points include sewing product key points and sewing line key points; and training the key point detection model by using the key point marked images. That is, the key points of the sewing product and the sewing line are designed according to the appearance features of the sewing product and the sewing line in the image. The sewing product key points mainly refer to the upper left corner point, the upper right corner point, the lower left corner point, the lower right corner point of the sewing product, and a plurality of equally divided points on the upper boundary of the sewing product. The sewing line key points mainly refer to the left boundary point, the right boundary point of the sewing line, and a plurality of equally divided points on the sewing line. Obviously, the key points of the sewing product and the sewing line have corresponding contour boundaries in the image, and the contour boundaries are composed of a series of pixel points in the image. Therefore, the key pixel points in the series of pixel points are regarded as the key points of the sewing product and the sewing line in the image. As can be seen from the above description, at least four points (i.e., the upper left corner point of the sewing product, the upper right corner point of the sewing product, the left boundary point of the sewing line, and the right boundary point of the sewing line) can locate the positions of the sewing product and the sewing line, and the more key points, the better the positioning effect.

[0058] For a better understanding, 20 key points of the sewing line of the sewing product are taken as an example for illustration, as shown in the following table: Figure 2 Specifically, the 9 key points 0-8 of the upper boundary of the sewing product in the image, the 9 key points 9-17 of the boundary of the sewing line in the image, the key point 18 of the lower left corner point of the sewing product in the image, and the key point 19 of the lower right corner point of the sewing product in the image. In particular, the key point 0 of the upper left corner point of the sewing product in the image, the key point 8 of the upper right corner point of the sewing product in the image, the key point 9 of the left boundary point of the sewing line in the image, and the key point 17 of the right boundary point of the sewing line in the image. Further, if the key points of the sewing product and the sewing line exist, all the key points in the image are uniquely represented by different coordinates (x, y), and if the key points of the sewing product or the sewing line do not exist, the corresponding key points in the image are represented by the origin coordinates (0, 0), that is, the key points of the non-sewing product key points and the risk key points in the image are represented by the origin coordinates (0, 0).

[0059] In this embodiment, after obtaining sufficient and accurately labeled sewing product seam key point data, the data can be trained using AlphaPose to obtain a key point detection model for detecting sewing product key points and seam key points, providing accurate positioning function of sewing products and seams for subsequent operations. It can be seen that for sewing product detection, the traditional edge detection algorithm commonly used in the prior art has the problem of being affected by the background and pattern, which can cause large positioning deviation. The AlphaPose key point detection of the sewing product seam is used to replace the traditional algorithm to improve the positioning accuracy and generalization performance of the woven bag and the seam.

[0060] Step S12: extracting an ROI region image from the initial image according to the detected key points.

[0061] In this embodiment, the sewing product is located from the initial image according to the detected key points, so as to extract the ROI region (region of interest) image corresponding to the sewing product. That is, before defect detection, the ROI region image of the woven bag can be intercepted. This is because the initial image contains a part of background information, and the background is redundant information for algorithm judgment. For example, as shown in the example in Figure 2 In the example, when there is a woven bag in the image, the redundant background information is located in the upper part of the image, and the woven bag is located in the middle and lower part of the image, so the sewing product region in the lower part of the image is the region where the ROI region image is located.

[0062] In this embodiment, the extracting an ROI region image from the initial image according to the detected key points can include: extracting an ROI region from the initial image according to a region selection rule; the region selection rule is that taking the sewing product boundary as the starting point to select a target height in the sewing product direction as the region height, and taking the initial image width as the region width; the sewing product boundary is determined according to the sewing product key points. It can be understood that under the premise that the key point detection model can provide accurate positioning of the sewing product and the seam, the ROI region is contained in the region from the sewing product boundary to the image lower boundary in the sewing product direction, and the width is the image width. Therefore, by taking the sewing product boundary as the starting point to select a target height in the sewing product direction as the region height, and taking the initial image width as the region width, the length and width of the ROI region image are obtained.

[0063] For example, in the example in Figure 2Generally, the captured image of the sewn product is an example, the ROI region of the sewn product is certainly included in the highest point of the upper boundary of the woven bag to the lowermost part of the image, that is, the up-down selection range of the ROI region can be the part of the image of the target height H selected downward from the highest point of the upper boundary of the woven bag, and the left-right selection range is the entire width of the image. It is particularly pointed out that the target height H must be greater than the difference between the lowest point of the seam boundary and the highest point of the upper boundary of the woven bag, and less than the difference between the image height and the highest point of the upper boundary of the woven bag. The target height H can be a reasonable value selected in combination with the data distribution of the historical sample image, and once the target height H is determined, it is also correspondingly determined in the inference process. The reasonable selection of the target height H helps to remove redundant background information. In addition, it should be noted that after the ROI image is intercepted, the coordinate position information of the key points needs to be correspondingly corrected.

[0064] Step S13: inputting the ROI region image into the stitching defect detection model to obtain the detection results output by each defect detection sub-model in the stitching defect detection model, and judging whether the to-be-detected sewn product has a stitching defect based on the detection results; the defect detection sub-model is a detection sub-model constructed according to the stitching defect type.

[0065] In this embodiment, the extracted ROI region image is input into the stitching defect detection model to obtain the detection results output by each defect detection sub-model in the stitching defect detection model, wherein the defect detection sub-model is a detection sub-model constructed according to the stitching defect type, that is, the stitching defect detection model is composed of at least two defect detection sub-models, and each defect detection sub-model is for a different defect type. After obtaining the detection results output by each defect detection sub-model, whether the sewn product has a stitching defect and the type of the defect if there is one are judged based on all the obtained detection results.

[0066] In this embodiment, the defect detection sub-models can each be composed of a deep learning feature extraction backbone network and a classification network layer, the feature extraction backbone network is composed of a plurality of convolution layers, BN layers, activation functions, pooling layers and global average pooling layers, and the classification network layer is composed of a plurality of fully connected layers and a Softmax function. At least one classification loss function is used in the training process, for example, a cross-entropy loss LossCE is used:

[0067]

[0068] wherein p is a prediction matrix of a sample, y is a real onehot label matrix of the sample, m is the number of samples, k is the number of categories, p ij is the prediction value corresponding to the jth category of the ith sample, y ijThe true onehot label value corresponding to the jth class of the ith sample. According to the data enhancement mode and the loss function described above, each defect detection sub-model is trained in turn until the algorithm stopping condition is met to end the training process.

[0069] In this embodiment, the defect detection sub-model construction process can include: obtaining normal images of the sewn product under normal sewing, and defect images under different sewing defect types; the sewing defect types include any two or more of the opening defect, the no-line defect, the broken-line defect, and the line-lapping defect; for each sewing defect type, a corresponding defect detection initial sub-model is constructed; training data corresponding to different sewing defect types is generated based on the normal images and the defect images; the defect detection initial sub-model of the corresponding type is trained using the training data corresponding to different sewing defect types to obtain the defect detection sub-model corresponding to different sewing defect types.

[0070] That is, before training the defect detection sub-model, the training data including training samples and test samples need to be obtained first. In this embodiment, the training samples and test samples are ROI region images obtained through the steps of "obtaining an initial image of a sewn product to be detected, detecting key points in the initial image using a key point detection model, and extracting a ROI region image from the initial image according to the detected key points". The training process of the sewing defect detection model is actually a classification task. The training data can be simply divided into two categories: normal samples (i.e. normal images) and defect samples (i.e. defect images). However, in order to improve the robustness of the model, the existing defect sample types are subdivided into N defect types, and N is an integer greater than or equal to 2. The N defect types include at least two types of opening, no-line, broken-line, and line-lapping. The opening refers to the simultaneous exposure of the inner side of the front layer surface and the inner side of the non-underfolding part of the back layer surface of the sewn product; the no-line refers to almost no stitching at the specified stitching of the sewn product; the broken-line refers to incomplete stitching at the specified stitching of the sewn product, including line interruption and line detachment; and the line-lapping refers to the stitching of the line only on the surface of the sewn product. The no-line, broken-line, and line-lapping are subdivisions of abnormal stitching, and are mutually exclusive; while the opening defect is not mutually exclusive with the abnormal stitching defect. Figure 3 In the above table, the first column represents the six types of defect samples, and the second column represents the corresponding defect types.

[0071] After obtaining the N defect type data, N defect detection initial sub-models are trained based on the defect types. For illustrative purposes, an example with N = 4 is described. If the first type of defect image is an opening image, the second type of defect image is a wireless image, the third type of defect image is a broken line image, and the fourth type of defect image is a line overlapping image. The normal image is mutually exclusive with the first type of defect image, the second type of defect image, the third type of defect image, and the fourth type of defect image. In this embodiment, the normal image in the training set is used to train four defect detection initial sub-models with the above four types of defect images in the training set. That is, the normal image and the first type of defect image are used to train the defect detection initial sub-model corresponding to the opening defect, the normal image and the second type of defect image are used to train the defect detection initial sub-model corresponding to the wireless defect, the normal image and the third type of defect image are used to train the defect detection initial sub-model corresponding to the broken line defect, and the normal image and the fourth type of defect image are used to train the defect detection initial sub-model corresponding to the line overlapping defect. Then, the first sub-model is used to distinguish the first type of defect image to obtain a first discrimination sub-result; the second sub-model is used to distinguish the second type of defect image to obtain a second discrimination sub-result; the third sub-model is used to distinguish the third type of defect image to obtain a third discrimination sub-result; and the fourth sub-model is used to distinguish the fourth type of defect image to obtain a fourth discrimination sub-result. The first discrimination sub-result, the second discrimination sub-result, the third discrimination sub-result, and the fourth discrimination sub-result are combined to determine whether the stitched image of the sewn product to be distinguished is a normal sample.

[0072] For example Figure 4 The woven bag stitching defect detection model integration reasoning process is shown, which includes obtaining an initial image of an object to be detected; a key point detection function module; obtaining an ROI image of a woven bag; integrating a stitching defect detection model of N defect type detection for comprehensive judgment; and determining whether the image is a normal sample or a defect sample. An example with N = 4 is described, i.e., there are four defect detection sub-models. Assuming that the discrimination results of the first sub-model, the second sub-model, the third sub-model, and the fourth sub-model for a normal sample are c1, c2, c3, and c4, respectively. Assuming that the specified threshold values of the first sub-model, the second sub-model, the third sub-model, and the fourth sub-model for a normal sample are T1, T2, T3, and T4, respectively. The final decision mode is: if c1 ≥ T1 and c2 ≥ T2 and c3 ≥ T3 and c4 ≥ T4, then the normal sample; otherwise, the defect sample.

[0073] In this embodiment, the model training of the initial sub-model for defect detection of the corresponding type of defect by using the training data corresponding to different types of stitching defects can include: if the type of stitching defect is a wireless defect, determining the suture interval segment between every two adjacent suture key points according to the suture key points on the ROI region image corresponding to the normal image; selecting a non-suture pixel region of the same size adjacent to the suture interval segment, and replacing the suture interval segment with the non-suture pixel region until all the suture interval segments are replaced to obtain a simulated wireless defect image; and using the normal image, the simulated wireless defect image and the wireless defect image as the training data corresponding to the wireless defect. It can be understood that, in order to improve the diversity of normal samples and defect samples, rich samples are designed through data enhancement. In the actual collection process of the stitching data, it is found that the proportion of defect samples is small, and therefore the diversity of negative samples is improved by simulating the wireless defect and the broken line defect. The core idea of simulating the wireless defect and the broken line defect is to first locate the suture position of the normal sample through key point detection, and then use cutting and pasting to select an appropriate pixel region from the non-suture region adjacent to the suture region, copy and paste it to the suture region, and complete the simulation of the defect.

[0074] The detailed execution steps of the wireless defect simulation can be as follows: (1) input the ROI image of the normal sample and the coordinate position of the key point from left to right on the suture boundary; (2) sequentially traverse the interval segment between every two adjacent suture key points from left to right, for each suture interval segment, copy and paste the pixel region of the same size adjacent to the suture interval segment below to the suture interval segment, cover the original suture pixel information, and traverse all the interval segments; (3) output the simulated wireless defect image and the corresponding category label. Through the above operation, at least the same amount of wireless defect samples as the normal samples can be simulated from the normal samples. In addition, since the pixel value of the covered suture is taken from the pixel value near the suture, the vertical texture after covering is basically consistent, and the horizontal segmentation of the suture according to the positioning point of the suture also ensures that the horizontal texture after covering is basically consistent, which is very close to the real situation. The above process can be executed online or offline by a computer program, which is efficient. Of course, it can also be manually processed, and the defect simulation effect will be more realistic, but manual processing is low in efficiency and high in time cost.

[0075] In the embodiment, the model training of the initial sub-model for defect detection of the corresponding type of defect by using the training data corresponding to different types of stitching defects can include: if the type of stitching defect is a broken line defect, determining the suture interval segment between every two adjacent suture key points on the ROI region image corresponding to the normal image, and randomly selecting part of the suture interval segments from all the suture interval segments as target suture interval segments; selecting a non-suture pixel region of the same size adjacent to the target suture interval segment, and replacing the target suture interval segment with the non-suture pixel region until all the target suture interval segments are replaced to obtain a simulated broken line defect image; and using the normal image, the simulated broken line defect image and the broken line defect image as the training data corresponding to the broken line defect. That is, by replacing part of the suture interval segments, the broken line simulation is realized, and at least the same amount of broken line defect samples as the normal samples can be simulated by the above operation.

[0076] The detailed execution steps of the broken line defect simulation can be as follows: (1) input the ROI image of the normal sample and the coordinate positions of the key points of the suture boundary from left to right. (2) every two adjacent suture key points can form a suture interval segment, and a plurality of interval segments are randomly selected from all the suture interval segments, the selected suture interval segments are traversed, and for each selected suture interval segment, the pixel region of the same size adjacent to the suture interval segment is copied and pasted to the suture interval segment, covering the original suture pixel information, until the selected interval segments are traversed. (3) output the simulated broken line defect image and the corresponding category label.

[0077] In the embodiment, the generation of the training data corresponding to different types of stitching defects based on the normal image and the defect image can include: randomly adding a watermark on the ROI region image of the normal image and randomly adding a watermark on the ROI region image of the defect image according to the watermarks in the watermark information library; and generating the training data corresponding to different types of stitching defects based on the normal image and the defect image after randomly adding the watermarks. It can be understood that in addition to the defect samples, in the process of solving the stitching defect detection, it is found that once there is an addition of the surface pattern of the stitched product, these normal samples will have a high false detection. In order to alleviate this problem, the random pattern on the surface of the simulated stitched product is enhanced by random watermarking, based on the fact that part of the images in the normal image after randomly adding the watermarks have watermarks and part of the images do not have watermarks, and the defect image is the same, thereby expanding the diversity of the surface patterns of the normal samples and the defect samples. The watermarking does not change the category label corresponding to the image.

[0078] The detailed execution steps of the watermark enhancement can be as follows: (1) constructing a watermark information library, including Arabic numeral symbols, English letter symbols, Chinese character symbols and other character symbols, punctuation symbols, mathematical symbols, and other printable symbols, etc. (2) inputting the ROI image of the sample and the corresponding sewing product and sewing line coordinates. (3) randomly selecting a plurality of symbols from the watermark information library. If the watermark is added in the upper part of the ROI image, a suitable small font size is set, and a suitable starting position of the watermark is selected according to the sewing product and the sewing line coordinates; if the watermark is added in the lower part of the ROI image, a suitable large or small font size is selected according to the set probability, and a suitable starting position of the watermark is selected according to the sewing product and the sewing line. Then, a random rotation angle and a transparency are selected. The watermark and the ROI image are fused according to the above parameters. (4) outputting the fused watermark enhanced image. For each defect detection initial sub-model corresponding to a type of sewing defect, the watermark enhancement can be used to increase the sample quantity, and the normal sample and the defect sample can be randomly used according to the probability.

[0079] As can be seen from the above, in the embodiment, an initial image of a to-be-detected sewing product is acquired, key point detection is performed on the initial image by using a key point detection model; an ROI region image is extracted from the initial image according to the detected key points; the ROI region image is input into a sewing defect detection model to obtain a detection result output by each defect detection sub-model in the sewing defect detection model, and whether the to-be-detected sewing product has a sewing defect is judged based on the detection result; the defect detection sub-model is a detection sub-model constructed according to a type of sewing defect. It can be seen that the sewing defect detection of the sewing product integrates the sewing line key point detection and the defect detection sub-models of multiple defect types as a whole, the positioning accuracy and the generalization performance are improved by positioning the sewing product through the key points, and the robustness of the sewing defect detection of the sewing product is improved by analyzing different defect types from multiple angles.

[0080] The embodiment of the present application discloses a specific sewing defect detection method, as shown in Figure 5 The method can include the following steps:

[0081] Step S21: acquiring an initial image of a to-be-detected sewing product, and performing key point detection on the initial image by using a key point detection model.

[0082] Step S22: determining whether the current image is empty according to the distance between each key point of the sewing product and the origin coordinate in combination with a first preset threshold value.

[0083] Step S23: determining whether the current image is a partial shot according to the horizontal distance between the left upper corner key point of the sewing product and the right upper corner key point of the sewing product in combination with a second preset threshold value.

[0084] In this embodiment, after determining the key points, the key points of the sewn product are used for filtering empty shooting and incomplete shooting. It can be understood that, on the packaging production line of the sewn product, the camera may miss shooting or shoot incompletely when shooting the image of the sewn product. In this case, the production line does not need to be stopped, but the next frame of image should be continuously shot for analysis and judgment. The detailed execution steps of the filtering of empty shooting and incomplete shooting based on the key points of the sewn product are as follows: it is judged whether the distances between all key points of the sewn product and the origin coordinate (0, 0) are less than a specified threshold value. If the condition is met, it is determined that the shooting is empty, and the key points of the non-sewn product and the risk key points in the image are represented by the origin coordinate (0, 0). Otherwise, it is judged whether the horizontal distance between the left upper corner point of the sewn product and the right upper corner point of the sewn product is greater than or equal to a specified threshold value. If the condition is not met, it is determined that the shooting is incomplete.

[0085] Step S24: whether the knitted product has a wireless defect is judged according to the distance between each stitch key point and the origin coordinate in combination with a third preset threshold value.

[0086] Step S25: whether the knitted product has a line overlapping defect or a line breaking defect is judged according to the horizontal distance between the stitch left boundary key point and the stitch right boundary key point in combination with a fourth preset threshold value.

[0087] In this embodiment, the key points of the stitches can also be used for coarse defect detection, that is, the key points of the stitches can be used to detect obvious stitch defects. That is, through preliminary analysis of the key points of the stitches, some relatively easy-to-distinguish wireless defects, line breaking defects or line overlapping defects can be reported. However, some defects such as line breaking in the middle and line overlapping from left to right cannot be accurately judged. In addition, due to the process deficiency of the sewn product in the packaging process, the sewn product may sometimes be tilted due to the pulling of some external force. At this time, the severely tilted sewn product needs to be reported by capturing the position of the stitches. The detailed execution steps of the above process are as follows: (1) it is judged whether the distances between the key points of the stitches and the coordinate (0, 0) are less than a specified threshold value. If the condition is met, it is determined that the sewn product has no wireless defect. Otherwise, it is judged whether the horizontal distance between the left boundary point of the stitches and the right boundary point of the stitches is greater than or equal to a specified threshold value. If the condition is not met, it is determined that the sewn product has a line breaking or line overlapping defect. If the condition is met, the tilt angle of the stitches is calculated through the left boundary point and the right boundary point of the stitches. If the tilt angle of the stitches is greater than the maximum value of the set interval, it is determined that the sewn product has a severe tilt defect. Otherwise, the next stage is entered.

[0088] In this embodiment, after the key point detection model is used to detect the key points of the initial image, the method can further include: determining the degree of inclination of the sewn product according to the inclination angle between the left boundary key point of the thread and the right boundary key point of the thread and a preset angle range; if the degree of inclination is adjustable, performing image correction on the initial image; if the degree of inclination is serious, determining that the initial image has a serious inclination defect. That is, for the woven bag that is inclined but not seriously inclined, image correction can be performed. Specifically, the inclination angle of the thread is calculated through the left boundary point and the right boundary point of the thread, and if the inclination angle of the thread is within the set interval, the image is rotated by the corresponding inclination angle for correction; if the inclination angle of the thread is less than the minimum value of the set interval, no image correction is performed.

[0089] Step S26: extracting an ROI region image from the initial image according to the detected key points.

[0090] Step S27: inputting the ROI region image into a stitching defect detection model to obtain a detection result output by each defect detection sub-model in the stitching defect detection model, and determining whether the to-be-detected sewn product has a stitching defect based on the detection result; the defect detection sub-model is a detection sub-model constructed according to a stitching defect type.

[0091] In this embodiment, the defect analysis based on the key point and the defect analysis based on the stitching defect detection model are combined, so that the defect detection efficiency is improved on the basis of improving the defect judgment accuracy and avoiding omission. The defect image detected through the key points can not be input into the stitching defect detection model.

[0092] The specific processes of the above steps S21, S22 and S26 can refer to the corresponding contents disclosed in the foregoing embodiments, which will not be described here in detail.

[0093] As can be seen from the above, in this embodiment, whether the current image is a blank shot is determined according to the distance between each key point of the sewn product and the origin coordinate in combination with a first preset threshold value; whether the current image is a partial shot is determined according to the horizontal distance between the upper left corner key point of the sewn product and the upper right corner key point of the sewn product in combination with a second preset threshold value; whether the woven product has a thread defect is determined according to the distance between each thread key point and the origin coordinate in combination with a third preset threshold value; whether the woven product has a thread overlapping defect or a thread breakage defect is determined according to the horizontal distance between the left boundary key point of the thread and the right boundary key point of the thread in combination with a fourth preset threshold value. The defect analysis based on the key point and the defect analysis based on the stitching defect detection model are combined, so that the defect detection efficiency is improved on the basis of improving the defect judgment accuracy and avoiding omission.

[0094] Correspondingly, the application further discloses a sewing defect detection device, which is shown in Figure 6 The device comprises:

[0095] a key point detection module 11, configured to acquire an initial image of a sewing product to be detected, and perform key point detection on the initial image by using a key point detection model;

[0096] a region extraction module 12, configured to extract an ROI region image from the initial image according to the detected key points;

[0097] a defect detection module 13, configured to input the ROI region image into a sewing defect detection model, obtain detection results output by each defect detection sub-model in the sewing defect detection model, and determine whether the sewing product to be detected has a sewing defect based on the detection results; the defect detection sub-model is a detection sub-model constructed according to a sewing defect type.

[0098] As can be seen, in the embodiment, an initial image of a sewing product to be detected is acquired, key point detection is performed on the initial image by using a key point detection model, an ROI region image is extracted from the initial image according to the detected key points, the ROI region image is input into a sewing defect detection model, detection results output by each defect detection sub-model in the sewing defect detection model are obtained, and it is determined whether the sewing product to be detected has a sewing defect based on the detection results; the defect detection sub-model is a detection sub-model constructed according to a sewing defect type. It can be seen that, for sewing defect detection of a sewing product, key point detection of a sewing thread and defect detection sub-models of multiple defect types are integrated as a whole, positioning accuracy and generalization performance of the sewing product are improved by key point positioning, different defect types are analyzed from multiple perspectives, and robustness of sewing defect detection of the sewing product is improved.

[0099] In some specific embodiments, the sewing defect detection device can specifically comprise:

[0100] a key point marking unit, configured to acquire a plurality of historical images containing a sewing product, add corresponding key points on the historical images according to a sewing product key point marking rule and a sewing thread key point marking rule, to obtain key point marked images; the key points include sewing product key points and sewing thread key points;

[0101] a key point detection model acquisition unit, configured to train the key point detection model by using the key point marked images.

[0102] In some specific embodiments, the sewing defect detection device can specifically comprise:

[0103] The empty image judgment unit is configured to judge whether the current image is empty according to a distance between each key point of the sewing product and the origin coordinate in combination with a first preset threshold value;

[0104] The incomplete capture judgment unit is configured to judge whether the current image is incomplete according to a horizontal distance between the upper-left key point of the sewing product and the upper-right key point of the sewing product in combination with a second preset threshold value;

[0105] The wireless defect judgment unit is configured to judge whether the knitted product has a wireless defect according to a distance between each key point of the sewing line and the origin coordinate in combination with a third preset threshold value;

[0106] The broken line defect judgment unit is configured to judge whether the knitted product has a broken line defect or a line overlapping defect according to a horizontal distance between the left boundary key point of the sewing line and the right boundary key point of the sewing line in combination with a fourth preset threshold value;

[0107] The inclination degree determination unit is configured to judge an inclination degree of the sewing product according to an inclination angle between the left boundary key point of the sewing line and the right boundary key point of the sewing line and a preset angle range;

[0108] The image correction unit is configured to perform image correction on the initial image if the inclination degree is an adjustable degree.

[0109] The inclination defect determination unit is configured to determine that the initial image has a serious inclination defect if the inclination degree is a serious inclination.

[0110] In some embodiments, the sewing defect detection device can specifically include:

[0111] The image acquisition unit is configured to acquire a normal image of the sewing product under normal sewing and defect images under different sewing defect types; the sewing defect types include any two or more of an opening defect, a wireless defect, a broken line defect and a line overlapping defect;

[0112] The initial sub-model construction unit is configured to construct a corresponding defect detection initial sub-model for each sewing defect type;

[0113] The training data generation unit is configured to generate training data corresponding to different sewing defect types based on the normal image and the defect images;

[0114] The sub-model training unit is configured to perform model training on the defect detection initial sub-model of the corresponding type by using the training data corresponding to different sewing defect types, so as to obtain defect detection sub-models corresponding to different sewing defect types.

[0115] In some embodiments, the sub-model training unit can specifically include:

[0116] The suture interval segment determination unit is configured to, if the suture defect type is a wire defect, determine a suture interval segment between each two adjacent suture key points according to the suture key points on the ROI region image corresponding to the normal image;

[0117] The replacement unit is configured to select a non-suture pixel region of the same size adjacent to the suture interval segment, and replace the suture interval segment with the non-suture pixel region, until all the suture interval segments are replaced to obtain a simulated wire defect image.

[0118] The training data determination unit is configured to take the normal image, the simulated wire defect image and a wire defect image as training data corresponding to the wire defect.

[0119] In some embodiments, the sub-model training unit can specifically include:

[0120] The target suture interval segment determination unit is configured to, if the suture defect type is a broken line defect, determine a suture interval segment between each two adjacent suture key points according to the suture key points on the ROI region image corresponding to the normal image, and randomly select part of the suture interval segments as target suture interval segments from all the suture interval segments.

[0121] The replacement unit is configured to select a non-suture pixel region of the same size adjacent to the target suture interval segment, and replace the target suture interval segment with the non-suture pixel region, until all the target suture interval segments are replaced to obtain a simulated broken line defect image.

[0122] The training data determination unit is configured to take the normal image, the simulated broken line defect image and a broken line defect image as training data corresponding to the broken line defect.

[0123] In some embodiments, the training data generation unit can specifically include:

[0124] The watermark adding unit is configured to randomly add a watermark on the ROI region image of the normal image and randomly add a watermark on the ROI region image of the defect image according to the watermarks in the watermark information library.

[0125] The training data determination unit is configured to generate training data corresponding to different suture defect types based on the normal image and the defect image after randomly adding the watermarks.

[0126] Further, the embodiments of the present application also disclose an electronic device, referring to Figure 7 The contents in the figure cannot be considered as any limitation on the use range of the present application.

[0127] Figure 7A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, the computer program is loaded and executed by the processor 21 to implement the related steps in the suture defect detection method disclosed in any of the foregoing embodiments.

[0128] In the embodiments, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0129] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon include an operating system 221, a computer program 222 and data 223 including an initial image, etc., and the storage mode can be temporary storage or permanent storage.

[0130] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the processor 21 on the mass data 223 in the memory 22, and the operating system 221 can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs for completing other specific work in addition to the computer programs for completing the suture defect detection method executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0131] Further, the embodiments of the present application further disclose a computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions are loaded and executed by the processor to implement the steps of the suture defect detection method disclosed in any of the foregoing embodiments.

[0132] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0133] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The term "computer-readable storage medium" includes, but is not limited to, non-transitory, tangible, and non-transitory, tangible media that store data for

[0134] Finally, it should be noted that, in the present document, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Additionally, unless otherwise stated, the term "about" preceding a value or description is understood in accordance with the phrase "within an acceptable manufacturing tolerance of" that value or description.

[0135] The above provides a detailed description of the stitching defect detection method, device, equipment and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in the present document. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A method of detecting a stitching defect, characterized by, The method comprises the following steps: acquiring an initial image of a sewing product to be detected, and performing key point detection on the initial image by using a key point detection model; determining whether the knitted product has a missing stitch defect according to the distance between each stitch key point and the origin coordinate in combination with a third preset threshold value; determining whether the knitted product has a line overlapping defect or a line breaking defect according to the horizontal distance between the left boundary key point of the stitch and the right boundary key point of the stitch in combination with a fourth preset threshold value; if the initial image does not have a missing stitch defect, a line overlapping defect or a line breaking defect, extracting an ROI region image from the initial image according to the detected key points; the key points include sewing product key points and stitch key points; inputting the ROI region image into a sewing defect detection model to obtain the detection results output by each defect detection sub-model in the sewing defect detection model, and determining whether the sewing product to be detected has a sewing defect based on the detection results; the sewing defect detection model comprises at least two defect detection sub-models, and each defect detection sub-model is constructed according to a sewing defect type; the defect detection sub-model is composed of a deep learning feature extraction backbone network and a classification network layer; wherein, the defect detection sub-model construction process comprises: acquiring a normal image of a sewing product under normal sewing and defect images under different sewing defect types; the sewing defect types include any two or more of the opening defect, the missing stitch defect, the line breaking defect and the line overlapping defect; constructing a corresponding defect detection initial sub-model for each sewing defect type; generating training data corresponding to different sewing defect types based on the normal image and the defect image; training the defect detection initial sub-model of the corresponding type by using the training data corresponding to different sewing defect types to obtain the defect detection sub-model corresponding to different sewing defect types.

2. The stitch defect detection method of claim 1, wherein, the key point detection model construction process comprises: acquiring a plurality of historical images containing sewing products, adding corresponding key points on the historical images according to a sewing product key point marking rule and a stitch key point marking rule to obtain a key point marked image; training the key point detection model by using the key point marked image.

3. The stitch defect detection method of claim 1, wherein, after the key point detection model is used to perform key point detection on the initial image, the method further comprises the following steps: determining whether the current image is empty according to the distance between each sewing product key point and the origin coordinate in combination with a first preset threshold value; determining whether the current image is a partial shot according to the horizontal distance between the upper left corner key point of the sewing product and the upper right corner key point of the sewing product in combination with a second preset threshold value; determining the inclination degree of the sewing product according to the inclination angle between the left boundary key point of the stitch and the right boundary key point of the stitch and a preset angle range; if the inclination degree is an adjustable degree, performing image correction on the initial image; if the inclination degree is a serious inclination, determining that the sewing product to be detected in the initial image has a serious inclination defect.

4. The stitch defect detection method of claim 1, wherein, The model training of the initial sub-model of the corresponding type of defect detection is performed by using the training data corresponding to different types of stitching defects, to obtain the defect detection sub-model corresponding to different types of stitching defects, and the method comprises the following steps: If the stitching defect type is a wireless defect, the suture interval segment between each two adjacent suture key points is determined according to the suture key points on the ROI region image corresponding to the normal image; A non-suture pixel region of the same size adjacent to the suture interval segment is selected, and the non-suture pixel region is used to replace the suture interval segment, until the simulated wireless defect image is obtained after all the suture interval segments are replaced; The normal image, the simulated wireless defect image and the wireless defect image are used as the training data corresponding to the wireless defect.

5. The stitch defect detection method of claim 1, wherein, The model training of the initial sub-model of the corresponding type of defect detection is performed by using the training data corresponding to different types of stitching defects, to obtain the defect detection sub-model corresponding to different types of stitching defects, and the method comprises the following steps: If the stitching defect type is a broken line defect, the suture interval segment between each two adjacent suture key points is determined according to the suture key points on the ROI region image corresponding to the normal image, and part of the suture interval segments are randomly selected as target suture interval segments from all the suture interval segments; A non-suture pixel region of the same size adjacent to the target suture interval segment is selected, and the non-suture pixel region is used to replace the target suture interval segment, until the simulated broken line defect image is obtained after all the target suture interval segments are replaced; The normal image, the simulated broken line defect image and the broken line defect image are used as the training data corresponding to the broken line defect.

6. The stitch defect detection method of claim 1, wherein, The training data corresponding to different types of stitching defects is generated based on the normal image and the defect image, and the method comprises the following steps: Watermarks are randomly added to the ROI region image of the normal image and the ROI region image of the defect image based on the watermarks in the watermark information library; The training data corresponding to different types of stitching defects is generated based on the normal image and the defect image after the watermarks are randomly added.

7. A stitching defect detection apparatus characterized by, It comprises: A key point detection module is configured to acquire an initial image of a sewing product to be detected, and detect key points in the initial image by using a key point detection model; Whether the woven product has a wireless defect is determined according to the distance between each suture key point and the origin coordinate in combination with a third preset threshold value, and whether the woven product has a line lifting defect or a broken line defect is determined according to the horizontal distance between the suture left boundary key point and the suture right boundary key point in combination with a fourth preset threshold value; A region extraction module is configured to extract an ROI region image from the initial image according to the detected key points if the initial image does not have a wireless defect, a line lifting defect or a broken line defect, wherein the key points include sewing product key points and suture key points; A defect detection module is configured to input the ROI region image into a stitching defect detection model to obtain detection results output by each defect detection sub-model in the stitching defect detection model, and determine whether the sewing product to be detected has a stitching defect based on the detection results. The stitching defect detection model comprises at least two defect detection sub-models, which are detection sub-models constructed according to stitching defect types, and each defect detection sub-model is for a different defect type; The defect detection sub-model is composed of a deep learning feature extraction backbone network and a classification network layer; The defect detection sub-model construction process comprises: obtaining normal images of a sewn product under normal stitching, and defect images under different stitching defect types; the stitching defect types include any two or more of opening defects, wireless defects, broken line defects and line overlapping defects; for each stitching defect type, a corresponding defect detection initial sub-model is constructed; training data corresponding to different stitching defect types is generated based on the normal images and the defect images; the defect detection initial sub-model of the corresponding type is trained by using the training data corresponding to different stitching defect types, to obtain the defect detection sub-model corresponding to different stitching defect types.

8. An electronic device, comprising: Comprise: a memory for saving a computer program; a processor for executing the computer program to implement the stitching defect detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, for storing a computer program; wherein the computer program is executed by a processor to implement the stitching defect detection method according to any one of claims 1 to 6.

Images