A label defect detection method and system based on image processing

Abstract

The present application relates to the field of image processing technology, and discloses a label defect detection method and system based on image processing. The method includes: constructing a target recognition model based on a label design drawing, extracting a label area image from a physical image based on the target recognition model, and aggregating the label area images contained in all commodities to generate a label image set; marking the category information of the label design drawing, inputting both the label image set and the label design drawing into a classification model, and the classification model performs multi-layer classification on the label image set based on the label design drawing to generate multiple classification results; based on the classification results, the label image set is divided into a standard label set and a non-standard label set, and any label area image in the non-standard label set is set as a defective label, and the evaluation value of the defective label is calculated. The defect category of the defective label is generated based on the evaluation value, and the defect detection analysis of the physical image is completed based on the defect category. The present application improves the efficiency and accuracy of label defect detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a label defect detection method and system based on image processing. Background Art

[0002] In an automated labeling production line, the quality of labels directly affects the product's market image and consumer satisfaction. Traditional methods of label defect detection rely mainly on manual visual inspection, which is inefficient, costly, and easily affected by human factors, resulting in unstable detection results. With the development of industrial automation technology, the demand for automated, high-precision detection of label defects is growing.

[0003] Existing automated inspection systems usually include sensors and mechanical devices that can perform basic label position and size detection, but have limited detection capabilities for subtle defects such as label contamination, wrinkling, and adhesive failure. The existence of these problems not only reduces production efficiency, but may also cause substandard goods to flow into the market. Therefore, by combining advanced image processing technology and machine learning algorithms, it is possible to automatically identify and classify various label defects, including but not limited to contamination, wrinkling, and adhesive failure.

[0004] Similar prior art includes a Chinese patent application with publication number CN117474924A, which discloses a label defect detection method based on machine vision, which includes: obtaining a label image of a label to be detected, and preprocessing the label image; respectively extracting contours of the target label image and the pre-acquired target template image; preliminarily matching the contour of the label to be detected with each template label contour in the template label contour set; classifying the contour of the label to be detected in the set of label contours to be detected; performing refinement matching on the contour of the label to be refined and the set of contours to be matched corresponding to the contour of the label to be refined; and generating label defect information corresponding to the label to be detected. This invention solves the technical problem of low efficiency and accuracy of defect detection of labels by performing image processing on label images, improves the efficiency and accuracy of defect detection of labels, and is mainly used for defect detection of labels.

[0005] The main deficiency of the existing technology is that it only detects defects on the label outline. In actual situations, labels also have defects such as missing content and format errors, which require accurate and complete defect detection of labels. Summary of the invention

[0006] The present application provides a label defect detection method and system based on image processing, which are used to improve the efficiency and accuracy of label defect detection based on image processing.

[0007] In a first aspect, the present application provides a label defect detection method based on image processing, and the label defect detection method based on image processing includes:

[0008] Obtain a label design corresponding to the product, build a target recognition model based on the label design, obtain a physical image of the product based on a preset shooting method, extract a label area image from the physical image based on the target recognition model, and aggregate the label area images contained in all products to generate a label image set;

[0009] Marking category information on the label design drawing, building a classification model, inputting both the label image set and the label design drawing into the classification model, and the classification model performing multi-layer classification on the label image set based on the label design drawing to generate multiple classification results;

[0010] The construction of the classification model comprises the following steps:

[0011] A first classification model is constructed based on a convolutional neural network, the first classification model is trained based on the label design drawing, feature labels are set, feature vectors of the label design drawing are extracted based on the feature labels, and the feature vectors are mapped to feature space, the feature space is divided into multiple subspaces based on the category information, a test set is extracted from the label design drawing, the mapping accuracy of the feature vectors in the subspace is verified based on the test set, the training parameters of the classification model are adjusted based on the mapping accuracy, the adjusted first classification model is set as the classification model, and the category information in the subspace is set as the type label of the subspace, wherein the feature label includes an edge label and a format label;

[0012] Based on the classification result, the label image set is divided into a standard label set and a non-standard label set, any label area image in the non-standard label set is set as a defect label, an evaluation value of the defect label is calculated, a defect category of the defect label is generated based on the evaluation value, and defect detection analysis of the entity image is completed based on the defect category.

[0013] In combination with the first aspect, extracting the label area image comprises the following steps:

[0014] Marking position information on the label design drawing to generate a marked image;

[0015] A recognition model is constructed based on a multi-layer neural network, all the annotated images are pre-processed and then input into the recognition model for training, the recognition model extracts image features of the annotated images, and marks pixel areas corresponding to the image features as recognition areas, the recognition areas and the annotated images are verified based on the position information and a first degree of overlap is output, model parameters of the recognition model are adjusted based on the first degree of overlap, this step is repeated until the first degree of overlap is greater than or equal to a first preset value, and the adjusted recognition model is set as the target recognition model;

[0016] The entity image is input into the target recognition model, the target recognition model recognizes the entity image based on the image features, sets the area of ​​the entity image containing the image features as the target area, sets the remaining area as the area to be refined, marks the target area with a target label, and marks the area to be refined with a label to be refined;

[0017] The entity image is segmented based on the target label and the label to be refined, and a target region map and a background region map are generated respectively, and the target region map is set as the label region image.

[0018] In combination with the first aspect, performing image segmentation on the entity image based on the target label and the label to be refined includes the following steps:

[0019] Setting a pixel selection box, wherein the pixel selection box slides in the entity image based on a preset selection method and generates a plurality of sub-areas, and setting the sub-area belonging to the target area as a first sub-area;

[0020] Extracting numerical features of the sub-regions, clustering all the numerical features to generate multiple clustering results, establishing region points based on the number of the sub-regions in the clustering results, obtaining image similarities between the sub-regions corresponding to any two region points based on the target labels, connecting all the region points based on the image similarities to generate a spliced ​​image, wherein the numerical features include pixel position information and pixel feature information;

[0021] The mosaic image containing the first sub-region is set as an extended mosaic image, the image similarity between the first sub-region and any of the sub-regions in the extended mosaic image is calculated and set as a first estimated value, if the first estimated value is greater than a second preset value, any pixel in the sub-region is marked with the target label, otherwise, any pixel in the sub-region is marked as a non-target label, and this step is repeated until all the sub-regions in the extended mosaic image are evaluated;

[0022] The entity image is pixel-cut based on the target label and the adjacent non-target labels, and all pixels corresponding to the target labels are set as the target area map, and all pixels corresponding to the non-target labels are set as the background area map.

[0023] In combination with the first aspect, generating a spliced ​​graph comprises the following steps:

[0024] Setting any of the sub-regions in the clustering result as the region point, calculating the average feature of the region point, extracting two adjacent region points based on the pixel position information, obtaining a feature difference value between the two region points based on the average feature, setting an edge coefficient between the two region points based on the feature difference value, and repeating this step until all the region points are calculated;

[0025] A plurality of combined graphs are constructed based on the regional points and the edge coefficients, an optimization algorithm is set, a sum of all the edge coefficients in the combined graphs is calculated based on the optimization algorithm and set as a first connection value, and the combined graph corresponding to the minimized first connection value is set as the spliced ​​graph.

[0026] In combination with the first aspect, generating multiple classification results comprises the following steps:

[0027] The classification model extracts a region vector of any of the label region images in the label image set, compares the region vector with the feature vector, and extracts a common vector and a difference vector from the region vector;

[0028] All the label area images are classified based on the common vector to generate multiple first classification results, the first classification results are classified based on the difference vector to generate multiple second classification results, and the label area image included in the second classification result is set as the classification result.

[0029] In combination with the first aspect, generating a standard label set and a non-standard label set based on the second classification result includes:

[0030] Extracting a label area image from the second classification result and setting it as a detection image, if the difference vector does not exist in the detection image, respectively extracting a first subvector and a second subvector corresponding to the edge label and the format label from the common vector, extracting a subspace corresponding to the first subvector and the second subvector from the feature space and setting it as a classification space, setting the category information corresponding to the classification space as a standard label type, otherwise, setting the detection image as a non-standard label type;

[0031] Repeat this step until all the label area images in any of the second classification results are extracted, and combine the detection images corresponding to all the standard label types to generate the standard label set, and combine the detection images corresponding to all the non-standard label types to generate the non-standard label set.

[0032] In combination with the first aspect, the defect category for generating the defect label based on the evaluation value includes:

[0033] In the classification space corresponding to the defect label, the feature vectors corresponding to the edge label and the format label are respectively obtained and set as a first-category vector and a second-category vector, a second degree of coincidence between the first-category vector and the first sub-vector is calculated, a third degree of coincidence between the second-category vector and the second sub-vector is calculated, and the second degree of coincidence and the third degree of coincidence are accumulated and set as the evaluation value of the defect label;

[0034] The defect categories include shape defects and content defects. If the evaluation value is less than or equal to a third preset value, the defect category of the defect label is determined. If the second overlap is less than a fourth preset value, the defect category is determined to be the shape defect. If the third overlap is less than a fifth preset value, the defect category is determined to be the content defect.

[0035] In combination with the first aspect, all the label area images in the entity image are summarized based on the defect category to complete the defect detection analysis.

[0036] In a second aspect, the present application provides a label defect detection system based on image processing, and the label defect detection system based on image processing includes:

[0037] A label extraction module is used to obtain a label design drawing corresponding to a product, build a target recognition model based on the label design drawing, obtain a physical image of the product based on a preset shooting method, extract a label area image from the physical image based on the target recognition model, and aggregate the label area images contained in all products to generate a label image set;

[0038] An image classification module is used to mark the category information of the label design drawing, build a classification model, input the label image set and the label design drawing into the classification model, and the classification model performs multi-layer classification on the label image set based on the label design drawing to generate multiple classification results; a first classification model is built based on a convolutional neural network, the first classification model is trained based on the label design drawing, a feature label is set, a feature vector of the label design drawing is extracted based on the feature label, and the feature vector is mapped to a feature space, the feature space is divided into multiple subspaces based on the category information, a test set is extracted from the label design drawing, and the mapping accuracy of the feature vector in the subspace is verified based on the test set, and the training parameters of the classification model are adjusted based on the mapping accuracy, the adjusted first classification model is set as the classification model, and the category information in the subspace is set as the type label of the subspace, wherein the feature label includes an edge label and a format label;

[0039] A defect recognition module is used to divide the label image set into a standard label set and a non-standard label set according to the classification result, set any label area image in the non-standard label set as a defect label, calculate an evaluation value of the defect label, generate a defect category of the defect label based on the evaluation value, and complete defect detection and analysis of the entity image based on the defect category.

[0040] In the technical solution provided in the present application, firstly, by constructing a target recognition model to identify and extract the label area image in the entity image, a large number of label images can be quickly processed, which is suitable for label defect detection needs in large-scale production environments and improves the efficiency of label defect detection. Then, by constructing a classification model and performing multi-layer classification on the label image set, the subtle features of the label area image can be captured more comprehensively and accurately, the accuracy of defect detection can be improved, and the standard label and defect label can be effectively distinguished to avoid misjudging the standard label as a defect label. Finally, by calculating the second degree of overlap between the first-category vector and the first sub-vector, and the third degree of overlap between the second-category vector and the second sub-vector, the defect category of the defective label is determined, thereby achieving efficient, accurate and explainable label defect detection.

[0041] This application also extracts the test image, determines whether there is a difference vector, extracts the sub-vector and sets the classification space, and finally generates a standard label set and a non-standard label set. It can effectively classify standard labels and non-standard labels and provide more accurate information for label defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0043] Figure 1 This is a schematic diagram of an embodiment of a label defect detection method based on image processing in an embodiment of the present application;

[0044] Figure 2 This is a schematic diagram of an embodiment of a label defect detection system based on image processing in an embodiment of the present application. DETAILED DESCRIPTION

[0045] Embodiments of the present application provide a label defect detection method and system based on image processing. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In an embodiment of the present application, an embodiment of a label defect detection method based on image processing includes:

[0047] S101. Obtain a label design drawing corresponding to the product, build a target recognition model based on the label design drawing, obtain a physical image of the product based on a preset shooting method, extract a label area image from the physical image based on the target recognition model, and aggregate the label area images contained in all products to generate a label image set.

[0048] It is understandable that the execution subject of the present application may be a label defect detection system based on image processing, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0049] This application is applied to the entire process from product label design to product labeling, which can improve the quality pass rate of product shipments.

[0050] Specifically, obtain the label design drawing corresponding to the product from the product design department or relevant database. The label design drawing can be a vector image (such as SVG) or a high-resolution bitmap (such as PNG, JPEG). The label design drawing should contain the complete layout information of the label, including text, patterns, barcodes, QR codes and other elements. Construct a target recognition model that can identify the label area, which is used to extract the label area image from the entity image later, wherein the preset shooting method refers to shooting the product when the label is affixed to the product, and the entity image containing the label can be obtained. The preset shooting method includes but is not limited to a fixed camera position, angle and shooting distance to ensure the consistency of the image, and use standardized lighting conditions to avoid shadows and reflections affecting the image quality. The acquired entity image is input into the target recognition model to extract the label area image, wherein the label area image refers to the image area where the label exists. The label image set refers to the collection of label area images extracted from the corresponding entity images of various products.

[0051] S102, marking category information on the label design drawing, building a classification model, inputting both the label image set and the label design drawing into the classification model, and the classification model performs multi-layer classification on the label image set based on the label design drawing to generate multiple classification results.

[0052] Building a classification model involves the following steps:

[0053] A first classification model is constructed based on a convolutional neural network. The first classification model is trained based on a label design drawing, feature labels are set, feature vectors of the label design drawing are extracted based on the feature labels, and the feature vectors are mapped to feature space. The feature space is divided into multiple subspaces based on category information, a test set is extracted from the label design drawing, and the mapping accuracy of the feature vectors in the subspace is verified based on the test set. The training parameters of the classification model are adjusted based on the mapping accuracy, the adjusted first classification model is set as the classification model, and the category information in the subspace is set as the type label of the subspace, wherein the feature labels include edge labels and format labels.

[0054] Specifically, the category information refers to the ID label in the product corresponding to the label design drawing, wherein the label design drawings refer to label drawings that fully comply with the design specifications. The label design drawings can be annotated using image annotation tools (such as LabelImg). The classification model is a model that can perform multi-layer classification on a label image set, and a classification model can be constructed using a convolutional neural network (CNN), such as ResNet, VGG and other models. The label image set and the label design drawing are both input into the classification model. The classification model is first trained using the label design drawing, and then the label image set is classified using the trained classification model to generate multiple classification results, wherein the classification results are used to identify defect labels and generate defect categories.

[0055] Specifically, the first classification model refers to an untrained model structure built using a convolutional neural network. Using the label design drawing to train the first classification model can effectively learn and identify the features corresponding to the correct label. Among them, convolutional neural networks (CNNs) perform well in image classification tasks, such as ResNet, VGG, Inception, etc. The use of feature labels can enable the first classification model to perform supervised learning and improve the training accuracy of the first classification model. The edge label in the feature label refers to the edge feature used to identify the label, such as edge strength, edge direction, etc., and the format label refers to the format feature used to identify the label, such as text font, pattern style, color distribution, etc. By extracting the feature vector of the label design drawing based on the feature label, each feature vector can be accurately distinguished, which is conducive to improving the accuracy of subsequent label defect detection.

[0056] Since there may be multiple feature vectors, the feature space can be understood as a high-dimensional space, where each dimension represents a feature vector. Category information is used to distinguish the categories corresponding to different label design drawings. Therefore, according to the category information, a supervised learning method can be used to divide the feature vectors of the label design drawings corresponding to the same category information in the feature space to generate multiple subspaces. In addition, the subspace division method can also use a clustering algorithm (such as K-means) to cluster the feature vectors into different subspaces.

[0057] The test set is a data set used to verify the training performance of the first classification model. The label design diagram in the test set is input into the first classification model to obtain the predicted subspace, and then the predicted subspace is compared with the real subspace to calculate the mapping accuracy, wherein the mapping accuracy can be set by the accuracy of the first classification model, for example, the ratio of the number of samples corresponding to the feature vector in the correctly classified subspace to the total number of samples corresponding to all feature vectors. According to the mapping accuracy, the training parameters of the classification model are adjusted, for example, the learning rate, batch size, regularization parameters, etc. The grid search or random search method can be used to adjust the parameters so that the mapping accuracy is greater than the first threshold, and the adjusted first classification model is set as the classification model.

[0058] S103. Based on the classification result, the label image set is divided into a standard label set and a non-standard label set, any label area image in the non-standard label set is set as a defect label, an evaluation value of the defect label is calculated, a defect category of the defect label is generated based on the evaluation value, and defect detection analysis of the entity image is completed based on the defect category.

[0059] Specifically, the standard label set refers to a set of label area maps without defects, and the non-standard label set refers to a set of label area maps with defects. Therefore, any label area map in the non-standard label set is set as a defective label. The evaluation value is used to calculate the integrity of the missing label, which can be calculated based on the following factors: defect type, different types of defects have different degrees of impact on product quality; defect degree, different severity of defects of the same type will also affect the evaluation value. Therefore, a defect category is generated based on the evaluation value, where the defect category refers to the type of defect, such as defects in shape contours caused by damage, wrinkling, and adhesive failure, and defects in content format errors caused by contaminated printing. By summarizing the defect categories corresponding to all label area maps in the physical image, the defect detection analysis of the label can be completed.

[0060] In a specific embodiment, extracting the label area image includes the following steps:

[0061] (1) Mark the location information of the label design drawing and generate an annotated image.

[0062] (2) Constructing a recognition model based on a multi-layer neural network, pre-processing all annotated images and inputting them into the recognition model for training; the recognition model extracts image features of the annotated images, and marks the pixel area corresponding to the image features as the recognition area; based on the position information, the recognition area and the annotated image are verified and a first degree of overlap is output; based on the first degree of overlap, the model parameters of the recognition model are adjusted; this step is repeated until the first degree of overlap is greater than or equal to a first preset value; and the adjusted recognition model is set as the target recognition model.

[0063] (3) The entity image is input into the target recognition model. The target recognition model recognizes the entity image based on the image features. The area in the entity image containing the image features is set as the target area, and the remaining area is set as the area to be refined. The target area is marked with a target label, and the area to be refined is marked with a label to be refined.

[0064] (4) Perform image segmentation on the entity image based on the target label and the label to be refined, generate a target area map and a background area map respectively, and set the target area map as the label area image.

[0065] Specifically, the position information refers to the position coordinates and size of the label design when it is planned to be pasted on the product. The position information can be used to determine whether the label area map conforms to the pasting plan in the physical image, wherein the position information can be converted through pixel coordinates. Annotated images refer to label design drawings with position information.

[0066] The recognition model refers to an untrained multi-layer neural network model, for example, a multi-layer neural network model corresponding to CNN (convolutional neural network), wherein the recognition model can also adopt an encoder-decoder architecture. Preprocessing refers to image processing methods such as image pixel normalization and resizing. Image features include but are not limited to features such as color, texture, and shape, and the recognition area refers to a pixel area composed of pixel points from which image features are extracted. Since the recognition model may contain information of all pixels or only part of the pixels when extracting image features of the annotated image, it is necessary to measure the feature extraction performance of the recognition model through the first degree of overlap. Among them, the first degree of overlap can be calculated using the intersection over union (IoU), and the intersection over union G is calculated based on the first formula. The first formula is: , where A represents the recognition area, B represents the annotated image, the intersection area (A∩B) represents the number of pixels covered by the recognition area and the annotated image, and the union area (A∪B) represents the total coverage area of ​​the recognition area and the annotated image, including pixels in the overlapping and non-overlapping parts. If the first overlap is closer to the first preset value (for example, 0.9), the training effect of the recognition model is better. Therefore, the model parameters of the recognition model can be adjusted according to the size of the first overlap, until the first overlap corresponding to the recognition model is greater than or equal to the first preset value, where the model parameters include but are not limited to parameters such as the learning rate. The adjusted recognition model is set as the target recognition model.

[0067] By inputting the entity image into the trained target recognition model, the features of the entity image can be extracted again and identified and classified with the image features of the existing annotated image. If the features of the entity image contain image features, it means that the pixel area corresponding to the features of the entity image belongs to the label feature. Therefore, the pixel area is set as the target area. Otherwise, the remaining pixel area in the entity image is set as the area to be refined. The area to be refined refers to pixels that may be labels or backgrounds. In order to further analyze and identify the pixels of the area to be refined, the pixels of the target area and the area to be refined can be marked by setting the target label and the label to be refined.

[0068] Image segmentation is performed according to the label types corresponding to various pixels, and a target region image and a background region image can be generated by segmentation, wherein both the target region image and the background region image can be multiple. The target region image can be set as a label region image, wherein the label region image can also be multiple.

[0069] In a specific embodiment, performing image segmentation on an entity image based on a target label and a label to be refined includes the following steps:

[0070] (1) Setting a pixel selection box. The pixel selection box slides in the entity image based on a preset selection method and generates multiple sub-regions. The sub-region belonging to the target region is set as the first sub-region.

[0071] (2) Extract the numerical features of the sub-regions, cluster all the numerical features to generate multiple clustering results, establish region points based on the number of sub-regions in the clustering results, obtain the image similarity between the sub-regions corresponding to any two region points based on the target labels, connect all the region points based on the image similarity, and generate a spliced ​​image, where the numerical features include pixel position information and pixel feature information.

[0072] (3) The mosaic image containing the first sub-region is set as the extended mosaic image, and the image similarity between the first sub-region and any sub-region in the extended mosaic image is calculated and set as the first estimated value. If the first estimated value is greater than the second preset value, any pixel in the sub-region is marked with a target label; otherwise, any pixel in the sub-region is marked with a non-target label. This step is repeated until all sub-regions in the extended mosaic image are evaluated;

[0073] (4) The entity image is pixel-wise segmented based on the target label and the adjacent non-target labels, and all pixels corresponding to the target label are set as the target region map, and all pixels corresponding to the non-target label are set as the background region map.

[0074] Specifically, the pixel selection box refers to a frame used to filter the pixel area. For example, setting the pixel selection box to 9×9 means selecting a square area of ​​9×9 pixels in the entity image in a preset selection method, wherein the preset selection method can set a sliding window for scrolling selection, so that the number of overlapping pixels between any two adjacent sub-areas selected by scrolling is 9. The size of the pixel selection box can be set according to the number of pixels in the entity image. In this way, the entity image can be divided into pixel areas corresponding to multiple sub-areas. If the sub-area completely overlaps with the target area, that is, the sub-area belongs to the inside of the target area, the sub-area is set as the first sub-area, wherein the first sub-area is a subset corresponding to multiple sub-areas, and the first sub-area can also be multiple.

[0075] The feature corresponding to the pixel position information in the numerical feature refers to the feature corresponding to the position coordinate of the sub-region in the entity image, and the feature corresponding to the pixel feature information in the numerical feature refers to the feature corresponding to the color, texture, shape, etc. in the sub-region. Therefore, the numerical feature can clearly describe the various features of the sub-region. All numerical features can be clustered using a clustering algorithm, wherein the clustering algorithm includes but is not limited to K-means clustering, hierarchical clustering, etc., and all numerical features include numerical features corresponding to all sub-regions. Generate multiple clustering results, wherein each clustering result can contain multiple sub-regions, and each sub-region in the clustering result is set as a region point, so the number of region points is equal to the number of sub-regions in the clustering result. The splicing graph refers to a structural graph generated after connecting each region point based on image similarity. Among them, the similarity of the numerical features of the sub-regions corresponding to any two region points can be calculated using methods such as cosine similarity and Euclidean distance, and the similarity is weighted in combination with the pixel position information to generate image similarity. For example, the closer the pixel position information is to the pixel position corresponding to the target region, the greater the weight value.

[0076] The first sub-region and other sub-regions contained in the extended mosaic image indicate that there is a high possibility that all sub-regions in the extended mosaic image belong to the target area. Therefore, it is necessary to evaluate the continuity between the sub-regions in the extended mosaic image by comparing the similarity between the first sub-region and other sub-regions in the extended mosaic image, wherein the similarity can be set by calculating the image similarity through numerical features. If the first estimated value is greater than the second preset value, it means that there is a high similarity between the sub-region and the first sub-region, so that there is pixel continuity between the sub-region and the first sub-region. Therefore, all pixels contained in the sub-region are marked with a target label, indicating that the sub-region belongs to the target area map. Otherwise, the possibility that the sub-region belongs to the target area map is excluded. Therefore, all pixels in the sub-region are marked with non-target labels, indicating that the sub-region belongs to the background area map. In actual situations, there may be a problem that the boundary between the product label and other background images of the product packaging is difficult to be completely segmented and recognized. Through this application, the recognition accuracy of the label area image can be improved in the physical image corresponding to the product.

[0077] There are edge features between the pixels corresponding to the target label and the adjacent pixels of the non-target label. Therefore, by performing pixel edge cutting between the pixels corresponding to the target label and the adjacent pixels of the non-target label in sequence, an accurate target area map and a background area map of the non-target area can be segmented.

[0078] In a specific embodiment, generating a mosaic graph comprises the following steps:

[0079] (1) Set any sub-region in the clustering result as a region point, calculate the average feature of the region point, extract two adjacent region points based on the pixel position information, obtain the feature difference value between the two region points based on the average feature, set the edge coefficient between the two region points based on the feature difference value, and repeat this step until all region points are calculated.

[0080] (2) Construct multiple combined graphs based on regional points and edge coefficients, set an optimization algorithm, calculate the sum of all edge coefficients in the combined graph based on the optimization algorithm and set it as the first connection value, and set the combined graph corresponding to the minimized first connection value as the spliced ​​graph.

[0081] Specifically, the average feature refers to the average value of the numerical feature based on the number of pixels in the sub-region corresponding to the regional point. Two adjacent regional points refer to the two sub-regions whose pixel coordinates are closest to each other based on the pixel position information. The feature difference value refers to the difference between the features of the same type in the average features corresponding to the two regional points. For example, the same type includes but is not limited to color feature type, texture feature type, etc. The edge coefficient refers to the weight value of the edge connecting two regional points. The higher the edge coefficient, the less likely it is that the two regional points belong to the same region. For example, if the feature difference value connecting the two regional points N1 and N2 is 10, then the edge coefficient between the regional point N1 and the regional point N2 is 10. According to this step, the edge coefficients between all regional points in the clustering results can be calculated.

[0082] From the calculated edge coefficients, we know that different regional points have different connection modes, so multiple combination graphs can be linked. An optimization algorithm is set based on mathematical statistics or graph cut algorithm to calculate the sum of all edge coefficients in any combination graph, that is, the first connection value. The graph cut algorithm is an algorithm based on graph theory, which is used to divide the vertex set of a graph into two non-intersecting subsets, so that the vertices in the two subsets are as "dissimilar" as possible, while keeping the vertices within the subsets as "similar" as possible. The first connection value is minimized through the optimization algorithm, and the combination graph corresponding to the minimum first connection value is set as the spliced ​​graph.

[0083] In a specific embodiment, generating multiple classification results includes the following steps:

[0084] (1) The classification model extracts the region vector of any labeled region image in the labeled image set, compares the region vector with the feature vector, and extracts the common vector and difference vector from the region vector.

[0085] (2) Classifying all label area images based on the common vector to generate a plurality of first classification results, classifying the first classification results based on the difference vector to generate a plurality of second classification results, and setting the label area images included in the second classification results as the classification results.

[0086] Specifically, the trained classification model extracts features from each label region image in the label image set to generate an image feature vector, i.e., a region vector. The extracted region vector is compared with the feature vector corresponding to the predefined label design diagram in the classification model, and the similarity between the region vector and each feature vector can be calculated, for example, by using cosine similarity, Euclidean distance, and other methods for comparison. A common vector refers to the characteristic part shared by the region vector and the feature vector, indicating the similarities between the label region image and the category information corresponding to the feature vector. A difference vector refers to the characteristic part that is different from the region vector and the feature vector, indicating the difference between the category information corresponding to the label region image and the feature vector. The specific extraction method can use the vector subtraction method corresponding to the dot product method, the vector decomposition method corresponding to the principal component analysis, the deep learning method corresponding to the autoencoder, and the like.

[0087] The nearest neighbor algorithm can be used to compare the common vector with the feature vector, and the category with the highest similarity can be selected as the first classification result, where classification means that the classification model classifies all label area images into subspaces corresponding to each category information according to the common vector, so that the label area images contained in the first classification result are located to the category information corresponding to the label design drawing. Since there may be difference vectors in each label area image in the first classification result, a clustering algorithm is used to classify the first classification result based on the difference vector to generate a second classification result, where the label area images contained in the second classification result may be of the same category information and have the same defect type. Therefore, the final classification result can be used for subsequent defect detection analysis.

[0088] In a specific embodiment, generating a standard label set and a non-standard label set based on the second classification result includes the following steps:

[0089] (1) A label area image is extracted from the second classification result and set as the detection image. If there is no difference vector in the detection image, the first subvector and the second subvector corresponding to the edge label and the format label are extracted from the common vector respectively, and the subspace corresponding to the first subvector and the second subvector is extracted from the feature space and set as the classification space. The category information corresponding to the classification space is set as the standard label type. Otherwise, the detection image is set as the non-standard label type.

[0090] (2) Repeat this step until all label area images in any second classification result are extracted, and combine the detection images corresponding to all standard label types to generate a standard label set, and combine the detection images corresponding to all non-standard label types to generate a non-standard label set.

[0091] Specifically, if there is no difference vector in the detection image, it means that the detection image conforms to the feature vector corresponding to the label design diagram in the classification model. Therefore, it is necessary to locate the category information corresponding to the detection image through the edge label and the format label in the common vector, that is, the standard label type. Among them, the first sub-vector corresponds to the characteristic part of the edge label, such as edge strength, edge direction, etc., and the second sub-vector corresponds to the characteristic part of the format label, such as text font, pattern style, color distribution, etc. The classification space is a subspace defined by the first sub-vector and the second sub-vector, which is used for the subsequent classification of the detection image. Otherwise, there is a difference vector in the detection image, and therefore it belongs to a non-standard label type, and further defect analysis of the detection image of the non-standard label type is required in the future.

[0092] By repeating this step, all label area images can be preliminarily screened out into a subset of the standard label set and a subset of the non-standard label set.

[0093] In a specific embodiment, generating a defect category of a defect label based on an evaluation value includes the following steps:

[0094] (1) In the classification space corresponding to the defect label, the feature vectors corresponding to the edge label and the format label are obtained respectively and set as the first-category vector and the second-category vector. The second degree of coincidence between the first-category vector and the first sub-vector is calculated, and the third degree of coincidence between the second-category vector and the second sub-vector is calculated. The second degree of coincidence and the third degree of coincidence are accumulated and set as the evaluation value of the defect label.

[0095] (2) Defect categories include shape defects and content defects. If the evaluation value is less than or equal to the third preset value, the defect category of the defect label is determined. If the second overlap is less than the fourth preset value, the defect category is determined to be a shape defect. If the third overlap is less than the fifth preset value, the defect category is determined to be a content defect.

[0096] Specifically, the feature vectors contained in the classification space correspond to the feature labels. Therefore, the feature vectors can be divided into first-category vectors and second-category vectors according to the feature labels. The second coincidence degree refers to the similarity between the first-category vector and the first sub-vector, which can be calculated by cosine similarity. Similarly, the third coincidence degree can be calculated. The evaluation value is generated by the accumulation of the second coincidence degree and the third coincidence degree, and the defect label can be evaluated from the analysis perspective corresponding to the two feature labels.

[0097] Shape defects are measured by the vector corresponding to the edge label, and content defects are measured by the vector corresponding to the format label. If the evaluation value is less than or equal to the third preset value, it means that the defective label has obvious label defect problems. In order to improve the detection accuracy of defective labels, a larger third preset value needs to be set. If the second overlap is less than the fourth preset value and the third overlap is less than the fifth preset value, it means that the defective label has both shape defects and content defects.

[0098] In a specific embodiment, all label area images in the entity image are aggregated based on defect categories to complete defect detection analysis.

[0099] Specifically, defect detection is performed on each label area image in the entity image in turn, and the defect category is output, which can improve the efficiency and accuracy of label defect detection of commodities.

[0100] The above describes the label defect detection method based on image processing in the embodiment of the present application. The following describes the label defect detection system based on image processing in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of the label defect detection system based on image processing includes:

[0101] The label extraction module 201 is used to obtain the label design drawing corresponding to the product, build a target recognition model based on the label design drawing, obtain the physical image of the product based on a preset shooting method, extract the label area image in the physical image based on the target recognition model, and aggregate the label area images contained in all products to generate a label image set.

[0102] The image classification module 202 is used to mark the category information of the label design drawing, build a classification model, input the label image set and the label design drawing into the classification model, and the classification model performs multi-layer classification on the label image set based on the label design drawing to generate multiple classification results; build a first classification model based on a convolutional neural network, the first classification model is trained based on the label design drawing, set feature labels, extract feature vectors of the label design drawing based on the feature labels, and map the feature vectors to the feature space, divide the feature space into multiple subspaces based on the category information, extract a test set from the label design drawing, verify the mapping accuracy of the feature vector in the subspace based on the test set, adjust the training parameters of the classification model based on the mapping accuracy, set the adjusted first classification model as the classification model, and set the category information in the subspace as the type label of the subspace, wherein the feature labels include edge labels and format labels.

[0103] The defect recognition module 203 is used to divide the label image set into a standard label set and a non-standard label set according to the classification result, set any label area image in the non-standard label set as a defect label, calculate the evaluation value of the defect label, generate the defect category of the defect label based on the evaluation value, and complete the defect detection analysis of the entity image based on the defect category.

[0104] Through the collaborative cooperation of the above components, firstly, by constructing a target recognition model to identify and extract the label area image in the entity image, a large number of label images can be processed quickly, which is suitable for the label defect detection needs in large-scale production environments and improves the efficiency of label defect detection. Then, by constructing a classification model and performing multi-layer classification on the label image set, the subtle features of the label area image can be captured more comprehensively and accurately, the accuracy of defect detection can be improved, and the standard label and defect label can be effectively distinguished to avoid misjudging the standard label as a defect label. Finally, by calculating the second degree of overlap between the first-category vector and the first sub-vector, and the third degree of overlap between the second-category vector and the second sub-vector, the defect category of the defect label is determined, achieving efficient, accurate and explainable label defect detection.

[0105] This application also extracts the test image, determines whether there is a difference vector, extracts the sub-vector and sets the classification space, and finally generates a standard label set and a non-standard label set. It can effectively classify standard labels and non-standard labels and provide more accurate information for label defect detection.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0108] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A label defect detection method based on image processing, characterized in that: The label defect detection method based on image processing includes: Obtain a label design corresponding to the product, build a target recognition model based on the label design, obtain a physical image of the product based on a preset shooting method, extract a label area image from the physical image based on the target recognition model, and aggregate the label area images contained in all products to generate a label image set; Marking category information on the label design drawing, building a classification model, inputting both the label image set and the label design drawing into the classification model, and the classification model performing multi-layer classification on the label image set based on the label design drawing to generate multiple classification results; The construction of the classification model comprises the following steps: A first classification model is constructed based on a convolutional neural network, the first classification model is trained based on the label design drawing, feature labels are set, feature vectors of the label design drawing are extracted based on the feature labels, and the feature vectors are mapped to feature space, the feature space is divided into multiple subspaces based on the category information, a test set is extracted from the label design drawing, the mapping accuracy of the feature vectors in the subspace is verified based on the test set, the training parameters of the classification model are adjusted based on the mapping accuracy, the adjusted first classification model is set as the classification model, and the category information in the subspace is set as the type label of the subspace, wherein the feature label includes an edge label and a format label; Based on the classification result, the label image set is divided into a standard label set and a non-standard label set, any label area image in the non-standard label set is set as a defect label, an evaluation value of the defect label is calculated, a defect category of the defect label is generated based on the evaluation value, and defect detection analysis of the entity image is completed based on the defect category.

2. The label defect detection method based on image processing according to claim 1, characterized in that: Extracting the label area image includes the following steps: Marking position information on the label design drawing to generate a marked image; A recognition model is constructed based on a multi-layer neural network, all the annotated images are pre-processed and then input into the recognition model for training, the recognition model extracts image features of the annotated images, and marks pixel areas corresponding to the image features as recognition areas, the recognition areas and the annotated images are verified based on the position information and a first degree of overlap is output, model parameters of the recognition model are adjusted based on the first degree of overlap, this step is repeated until the first degree of overlap is greater than or equal to a first preset value, and the adjusted recognition model is set as the target recognition model; The entity image is input into the target recognition model, the target recognition model recognizes the entity image based on the image features, sets the area of ​​the entity image containing the image features as the target area, sets the remaining area as the area to be refined, marks the target area with a target label, and marks the area to be refined with a label to be refined; The entity image is segmented based on the target label and the label to be refined, and a target region map and a background region map are generated respectively, and the target region map is set as the label region image.

3. The label defect detection method based on image processing according to claim 2, characterized in that: Performing image segmentation on the entity image based on the target label and the label to be refined comprises the following steps: Setting a pixel selection box, wherein the pixel selection box slides in the entity image based on a preset selection method and generates a plurality of sub-areas, and setting the sub-area belonging to the target area as a first sub-area; Extracting numerical features of the sub-regions, clustering all the numerical features to generate multiple clustering results, establishing region points based on the number of the sub-regions in the clustering results, obtaining image similarities between the sub-regions corresponding to any two region points based on the target labels, connecting all the region points based on the image similarities to generate a spliced ​​image, wherein the numerical features include pixel position information and pixel feature information; The mosaic image containing the first sub-region is set as an extended mosaic image, the image similarity between the first sub-region and any of the sub-regions in the extended mosaic image is calculated and set as a first estimated value, if the first estimated value is greater than a second preset value, any pixel in the sub-region is marked with the target label, otherwise, any pixel in the sub-region is marked as a non-target label, and this step is repeated until all the sub-regions in the extended mosaic image are evaluated; The entity image is pixel-cut based on the target label and the adjacent non-target labels, and all pixels corresponding to the target labels are set as the target area map, and all pixels corresponding to the non-target labels are set as the background area map.

4. The label defect detection method based on image processing according to claim 3 is characterized in that: Generating a mosaic graph comprises the following steps: Setting any of the sub-regions in the clustering result as the region point, calculating the average feature of the region point, extracting two adjacent region points based on the pixel position information, obtaining a feature difference value between the two region points based on the average feature, setting an edge coefficient between the two region points based on the feature difference value, and repeating this step until all the region points are calculated; A plurality of combined graphs are constructed based on the regional points and the edge coefficients, an optimization algorithm is set, a sum of all the edge coefficients in the combined graphs is calculated based on the optimization algorithm and set as a first connection value, and the combined graph corresponding to the minimized first connection value is set as the spliced ​​graph.

5. The label defect detection method based on image processing according to claim 1, characterized in that: Generating multiple classification results comprises the following steps: The classification model extracts a region vector of any of the label region images in the label image set, compares the region vector with the feature vector, and extracts a common vector and a difference vector from the region vector; All the label area images are classified based on the common vector to generate multiple first classification results, the first classification results are classified based on the difference vector to generate multiple second classification results, and the label area image included in the second classification result is set as the classification result.

6. The label defect detection method based on image processing according to claim 5, characterized in that: Generating a standard label set and a non-standard label set based on the second classification result includes: Extracting a label area image from the second classification result and setting it as a detection image, if the difference vector does not exist in the detection image, respectively extracting a first subvector and a second subvector corresponding to the edge label and the format label from the common vector, extracting a subspace corresponding to the first subvector and the second subvector from the feature space and setting it as a classification space, setting the category information corresponding to the classification space as a standard label type, otherwise, setting the detection image as a non-standard label type; Repeat this step until all the label area images in any of the second classification results are extracted, and combine the detection images corresponding to all the standard label types to generate the standard label set, and combine the detection images corresponding to all the non-standard label types to generate the non-standard label set.

7. The label defect detection method based on image processing according to claim 6, characterized in that: The defect categories for generating the defect labels based on the evaluation values ​​include: In the classification space corresponding to the defect label, the feature vectors corresponding to the edge label and the format label are respectively obtained and set as a first-category vector and a second-category vector, a second degree of coincidence between the first-category vector and the first sub-vector is calculated, a third degree of coincidence between the second-category vector and the second sub-vector is calculated, and the second degree of coincidence and the third degree of coincidence are accumulated and set as the evaluation value of the defect label; The defect categories include shape defects and content defects. If the evaluation value is less than or equal to a third preset value, the defect category of the defect label is determined. If the second overlap is less than a fourth preset value, the defect category is determined to be the shape defect. If the third overlap is less than a fifth preset value, the defect category is determined to be the content defect.

8. The label defect detection method based on image processing according to claim 1, characterized in that: All the label area images in the entity image are aggregated based on the defect category to complete the defect detection analysis.

9. A label defect detection system based on image processing, characterized in that: The label defect detection system based on image processing comprises: A label extraction module is used to obtain a label design drawing corresponding to a product, build a target recognition model based on the label design drawing, obtain a physical image of the product based on a preset shooting method, extract a label area image from the physical image based on the target recognition model, and aggregate the label area images contained in all products to generate a label image set; An image classification module is used to mark the category information of the label design drawing, build a classification model, input the label image set and the label design drawing into the classification model, and the classification model performs multi-layer classification on the label image set based on the label design drawing to generate multiple classification results; a first classification model is built based on a convolutional neural network, the first classification model is trained based on the label design drawing, a feature label is set, a feature vector of the label design drawing is extracted based on the feature label, and the feature vector is mapped to a feature space, the feature space is divided into multiple subspaces based on the category information, a test set is extracted from the label design drawing, and the mapping accuracy of the feature vector in the subspace is verified based on the test set, and the training parameters of the classification model are adjusted based on the mapping accuracy, the adjusted first classification model is set as the classification model, and the category information in the subspace is set as the type label of the subspace, wherein the feature label includes an edge label and a format label; A defect recognition module is used to divide the label image set into a standard label set and a non-standard label set according to the classification result, set any label area image in the non-standard label set as a defect label, calculate an evaluation value of the defect label, generate a defect category of the defect label based on the evaluation value, and complete defect detection and analysis of the entity image based on the defect category.

Images