Product appearance defect detection system and method based on image semantics

Abstract

The invention discloses a product appearance flaw detection system and method based on image semanteme, and belongs to the technical field of flaw detection, and the system comprises an image collection module which is used for collecting a product appearance real-time image; the image processing module is used for processing the product appearance real-time image; the image semantic module is used for performing feature extraction and semantic segmentation on the product appearance real-time image to generate a product appearance semantic image; and the flaw detection module is used for performing flaw detection on the appearance of the product, judging whether the appearance of the product has flaws or not, and performing early warning, alarming and visual display on the appearance flaws of the product. The problem that the product appearance defect detection effect is poor due to the fact that effective defect detection cannot be carried out on the product appearance based on image semantics in the prior art is solved. According to the invention, effective flaw detection can be carried out on the product appearance based on image semantics, timely early warning can be carried out according to the product appearance flaw detection condition, and the product appearance flaw detection effect can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and specifically to a product appearance defect detection system and method based on image semantics. Background Art

[0002] Product appearance defect detection refers to detecting whether there are defective phenomena on the product surface through a series of technical means, which may cause the product not to meet the reasonable safety requirements necessary for consumption or use. Therefore, it is necessary to detect the appearance defects of the product.

[0003] Chinese Patent with Publication No. CN116883370B discloses an agricultural product appearance quality detection system, including an image filtering module for filtering the grayscale image corresponding to the appearance image of the agricultural product: calculating the filtering comparison value of the pixel points in the grayscale image; obtaining the pixel points belonging to noise in the grayscale image based on the filtering comparison value; calculating the range parameter of the pixel points belonging to noise; performing median filtering on the pixel points belonging to noise based on the range parameter to obtain a filtered image, enabling the filtering range to automatically change with the change of the attributes of the pixel points belonging to noise, greatly reducing the occurrence probability of the situation where the pixel values obtained after filtering are too large, facilitating obtaining a more accurate median filtering result, improving the accuracy of the obtained image features, and making the appearance quality detection result more accurate; however, this patent has the following defects:

[0004] The existing technologies cannot effectively detect product appearance defects based on image semantics and cannot give timely warnings according to the product appearance defect detection situation, resulting in poor product appearance defect detection effects. Summary of the Invention

[0005] The purpose of the present invention is to provide a product appearance defect detection system and method based on image semantics, which can effectively detect product appearance defects based on image semantics and can give timely warnings according to the product appearance defect detection situation, improving the product appearance defect detection effect and solving the problems raised in the above background art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] A product appearance defect detection system based on image semantics, comprising:

[0008] An image acquisition module for acquiring real-time images of the product appearance based on computer vision;

[0009] An image processing module for processing the real-time images of the product appearance based on computer vision;

[0010] An image semantic module for extracting features and performing semantic segmentation on the real-time image of the product appearance to generate a semantic image of the product appearance;

[0011] A defect detection module for detecting defects on the product appearance based on machine learning technology, determining whether there are defects on the product appearance, and giving early warning alarms and visual displays for the defect situation of the product appearance.

[0012] Preferably, the image acquisition module includes:

[0013] A product fixing unit for fixing the product to be detected;

[0014] Place the product to be detected on the product appearance defect detection station and fix the product to be detected to make the product to be detected placed stably;

[0015] An acquisition adjustment unit for adjusting the acquisition angle and light intensity of the product to be detected;

[0016] When acquiring the image of the product to be detected, adjust the acquisition angle and light intensity of the product to be detected based on an intelligent adjustment device;

[0017] A product acquisition unit for acquiring real-time images of the product appearance;

[0018] Based on computer vision technology, use machine vision equipment to perform real-time monitoring and continuous acquisition of the product to be detected, obtain product images of the product to be detected under different acquisition angles and light intensity conditions, and determine the real-time image of the product appearance based on computer vision.

[0019] Preferably, the acquisition adjustment unit further specifically performs the following steps:

[0020] Extract each light intensity initial value included in the light intensity initial value group for the product;

[0021] After the test product is fixed, take pictures of the test product according to each light intensity initial value to obtain product test images corresponding to each light intensity initial value;

[0022] Compare the gray values of the pixel points included in the product test images corresponding to each light intensity initial value with a preset gray threshold value to obtain the area of the target region formed by the pixel points whose gray values exceed the preset gray threshold value;

[0023] Obtain a light intensity adjustment coefficient by using the area of the target region of the product image corresponding to each light intensity initial value;

[0024] Wherein, the light intensity adjustment coefficient is obtained through the following formula:

[0025]

[0026] Among them, L represents the light intensity adjustment coefficient; n represents the number of initial light intensity values included in the initial light intensity group; B i represents the i-th initial light intensity value included in the initial light intensity group; B i+1 represents the (i + 1)-th initial light intensity value included in the initial light intensity group; S i represents the target area area of the product test image corresponding to the i-th initial light intensity value; S i+1 represents the target area area of the product test image corresponding to the (i + 1)-th initial light intensity value; B b represents the standard deviation of the n initial light intensity values included in the initial light intensity group; S b represents the standard deviation of the target area areas of the product test images corresponding to the n initial light intensity values included in the initial light intensity group;

[0027] Adjust the light intensity for image acquisition of the product to be detected by using the light intensity adjustment coefficient.

[0028] Preferably, adjusting the light intensity for image acquisition of the product to be detected by using the light intensity adjustment coefficient includes:

[0029] Retrieve the initial light intensity value corresponding to the minimum target area area of the product test image as the first light intensity;

[0030] Retrieve the initial light intensity value corresponding to the maximum target area area of the product test image as the second light intensity;

[0031] Obtain a light intensity reference value by using the first light intensity and the second light intensity;

[0032] Among them, the light intensity reference value is obtained through the following formula:

[0033]

[0034] Among them, B c represents the light intensity reference value; B p represents the average value of the initial light intensity values included in the initial light intensity group; B min represents the first light intensity; B max represents the second light intensity; n represents the number of initial light intensity values included in the initial light intensity group; H zi represents the gray middle value corresponding to the target area area corresponding to the i-th initial light intensity value; H ki represents the center gray value of the product test image corresponding to the i-th initial light intensity value;

[0035] Retrieve the light intensity adjustment coefficient;

[0036] Obtain the light intensity of the product to be detected by using the light intensity adjustment coefficient and the light intensity reference value;

[0037] Among them, the light intensity is obtained through the following formula:

[0038]

[0039] Among them, B t represents the light intensity; B c represents the light intensity reference value; L represents the light intensity adjustment coefficient.

[0040] Preferably, the image processing module includes:

[0041] An image denoising unit for denoising the real-time image of the product appearance based on computer vision;

[0042] Obtain the real-time image of the product appearance based on computer vision, convert the real-time image of the product appearance based on computer vision into a grayscale image, and perform denoising processing on the real-time image of the product appearance based on computer vision based on a median filter;

[0043] Among them, according to the noise type and intensity in the real-time image of the product appearance based on computer vision, as well as the image details and size, set the window size of the median filter;

[0044] When the noise in the real-time image of the product appearance based on computer vision is salt-and-pepper noise or random noise and the noise intensity is high, use a large window; when the noise intensity is low, use a small window;

[0045] When the image contains important detail or edge information, use a small window; when the image size is large, use a small window or adopt a block processing method;

[0046] After the window size of the median filter is set, for each pixel point in the real-time image of the product appearance based on computer vision, calculate the median of the grayscale values of all pixel points in its surrounding window-size neighborhood, replace the grayscale value of this pixel point with the median, for removing or reducing the noise in the real-time image of the product appearance based on computer vision, and save or display the denoised real-time image of the product appearance.

[0047] Preferably, the image processing module further includes:

[0048] An image adjustment unit for adjusting the brightness and contrast of the denoised real-time image of the product appearance;

[0049] Brightness adjustment: Obtain the grayscale values of all pixels in the real-time image of the product appearance after denoising. Increase the brightness by increasing the grayscale values of all pixels in the real-time image of the product appearance after denoising, and decrease the brightness by decreasing the grayscale values of all pixels in the real-time image of the product appearance after denoising;

[0050] Contrast adjustment: Based on histogram equalization, perform contrast adjustment on the real-time image of the product appearance after denoising. By redistributing the grayscale values of the real-time image of the product appearance, adjust the grayscale distribution of the real-time image of the product appearance to a uniform distribution, and increase the dynamic range of the pixel grayscale values to adjust the contrast of the real-time image of the product appearance.

[0051] Preferably, the image semantic module includes:

[0052] An extraction and matching unit for performing feature extraction and matching on the processed real-time image of the product appearance;

[0053] Based on a convolutional neural network deep learning model, perform feature extraction on the processed real-time image of the product appearance, automatically learn the features of the real-time image of the product appearance, extract highly discriminative feature vectors, including shape features, texture features, and color features, and use a similarity algorithm to match the extracted feature vectors with the features in the existing feature library;

[0054] A semantic segmentation unit for performing semantic segmentation on the real-time image of the product appearance, dividing the real-time image of the product appearance into different semantic regions, each semantic region having the same class label, and fusing the extracted feature vectors and the semantic segmentation results, splicing the feature vectors with the semantic labels to generate a unified semantic image of the product appearance.

[0055] Preferably, the defect detection module includes:

[0056] A model training unit for training a product appearance defect detection model based on image semantics;

[0057] According to the product appearance defect detection requirements based on image semantics, collect historical images of the product appearance;

[0058] Divide the collected historical images of the product appearance to determine the training set and the test set;

[0059] Based on the training set, train and iterate the machine learning model so that the machine learning model automatically learns product appearance defect detection to determine a product appearance defect detection model based on image semantics;

[0060] Based on the test set, perform performance testing on the product appearance defect detection model based on image semantics, determine whether the product appearance defect detection model based on image semantics can achieve the expected effect, and determine the optimal product appearance defect detection model based on image semantics.

[0061] Preferably, the defect detection module further includes:

[0062] A defect detection unit for detecting defects in the product appearance;

[0063] Obtain the optimal product appearance defect detection model based on image semantics;

[0064] Deploy the optimal product appearance defect detection model based on image semantics to the actual product appearance defect detection environment;

[0065] Perform image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics, check whether there are defects in the product appearance, and determine the product appearance defect detection result based on image semantics;

[0066] An early warning display unit for giving early warning alarms and visual display of the product appearance defect situation;

[0067] Obtain the product appearance defect detection result based on image semantics;

[0068] When it is detected that there are defects in the product appearance, automatically give an early warning alarm and notify the management personnel in time;

[0069] Based on the product information, mark the product appearance defect situation to form a product appearance defect detection report, and visually display the product appearance defect detection report to the management personnel in a visual form.

[0070] According to another aspect of the present invention, there is provided a product appearance defect detection method based on image semantics, which is implemented based on the product appearance defect detection system based on image semantics as described above, and includes the following steps:

[0071] S1: Place the product to be detected on the product appearance defect detection station, and based on computer vision technology, perform real-time acquisition of the product to be detected under different acquisition angles and illumination intensity conditions to determine the real-time product appearance image based on computer vision;

[0072] S2: Based on the median filter, perform denoising processing on the real-time product appearance image based on computer vision, remove or reduce the noise in the real-time product appearance image based on computer vision, and save or display the denoised real-time product appearance image;

[0073] S3: Adjust the brightness of the real-time product appearance image by increasing or decreasing the grayscale values of all pixels in the real-time product appearance image, and based on histogram equalization, increase the dynamic range of the pixel grayscale values in the real-time product appearance image to adjust the contrast of the real-time product appearance image;

[0074] S4: Extract features from the real-time product appearance image based on a convolutional neural network deep learning model to extract highly discriminative feature vectors, and use a similarity algorithm to match the extracted feature vectors with the features in the existing feature library, and perform semantic segmentation on the real-time product appearance image, and fuse the extracted feature vectors and the semantic segmentation results to generate a product appearance semantic image:

[0075] S5: Perform image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics to determine whether there are defects in the product appearance. When defects in the product appearance are detected, an early warning alarm is automatically issued to notify the management personnel in a timely manner, and based on the product information, mark the product appearance defect situation to form a product appearance defect detection report, and visually display the product appearance defect detection report to the management personnel in a visual form.

[0076] Compared with the prior art, the beneficial effects of the present invention are:

[0077] 1. Based on computer vision technology, the present invention performs real-time acquisition of the product to be detected under different acquisition angles and light intensity conditions to determine the real-time product appearance image based on computer vision. Based on the median filter, denoise the real-time product appearance image based on computer vision to remove or reduce the noise in the real-time product appearance image based on computer vision, and adjust the brightness and contrast of the denoised real-time product appearance image. Extract features from the real-time product appearance image based on a convolutional neural network deep learning model to extract highly discriminative feature vectors, and use a similarity algorithm to match the extracted feature vectors with the features in the existing feature library, and perform semantic segmentation on the real-time product appearance image, and fuse the extracted feature vectors and the semantic segmentation results to generate a product appearance semantic image, which is convenient for subsequent effective defect detection of the product appearance.

[0078] 2. The present invention performs image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics, determines whether there are defects in the product appearance. When defects are detected in the product appearance, an early warning alarm is automatically issued to notify the management personnel in a timely manner. Moreover, based on the product information, the product appearance defect situation is marked to form a product appearance defect detection report, and the product appearance defect detection report is visually displayed to the management personnel in a visual form, facilitating the management personnel to manage the products with defective appearances in a timely manner. It can effectively detect defects in the product appearance based on image semantics and can give an early warning in a timely manner according to the product appearance defect detection situation, improving the product appearance defect detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a module structure diagram of the product appearance defect detection system based on image semantics of the present invention;

[0080] Figure 2 It is an operation flowchart of the product appearance defect detection method based on image semantics of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0081] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0082] To solve the problem that the existing technology cannot effectively detect defects in the product appearance based on image semantics and cannot give an early warning in a timely manner according to the product appearance defect detection situation, resulting in poor product appearance defect detection effect, please refer to Figure 1 - Figure 2 , the following technical solutions are provided in this embodiment:

[0083] Embodiment 1

[0084] The product appearance defect detection system based on image semantics includes: an image acquisition module, an image processing module, an image semantics module, and a defect detection module.

[0085] It should be noted that through the interaction and communication between the image acquisition module, the image processing module, the image semantics module, and the defect detection module, defects in the product appearance can be effectively detected based on image semantics, and an early warning can be given in a timely manner according to the product appearance defect detection situation, improving the product appearance defect detection effect.

[0086] Specifically, the image acquisition module performs real-time acquisition on the product to be detected under different acquisition angles and illumination intensity conditions to determine the real-time image of the product appearance. The image processing module processes the real-time image of the product appearance, removes the noise in the real-time image of the product appearance, and adjusts the brightness and contrast of the real-time image of the product appearance. The image semantics module extracts features and performs semantic segmentation on the real-time image of the product appearance to generate a semantic image of the product appearance. The defect detection module performs defect detection on the product appearance, determines whether there are defects in the product appearance, and issues early warning alarms and visual displays for the defect situation of the product appearance.

[0087] In this embodiment, as a preferred technical solution of the present invention, the image acquisition module includes:

[0088] A product fixing unit for fixing the product to be detected;

[0089] Place the product to be detected on the product appearance defect detection station and fix the product to be detected to make the product to be detected placed stably, facilitating the acquisition of the appearance of the product to be detected.

[0090] An acquisition adjustment unit for adjusting the acquisition angle and illumination intensity of the product to be detected;

[0091] When acquiring the image of the product to be detected, adjust the acquisition angle and illumination intensity of the product to be detected based on the intelligent adjustment device, so as to facilitate the acquisition of the appearance image of the product to be detected under different acquisition angles and illumination intensity conditions.

[0092] A product acquisition unit for acquiring the real-time image of the product appearance;

[0093] Based on computer vision technology, use machine vision equipment to perform real-time monitoring and continuous acquisition on the product to be detected, obtain the product images of the product to be detected under different acquisition angles and illumination intensity conditions, and determine the real-time image of the product appearance based on computer vision.

[0094] It should be noted that computer vision refers to using cameras and computers to replace the human eye to perform machine vision such as target recognition, tracking, and measurement on targets, and further performing graphic processing to make the computer process into an image more suitable for human eye observation or transmission to an instrument for detection; while machine vision is to use a machine to replace the human eye for measurement and judgment. The machine vision system converts the captured target into an image signal through machine vision products, transmits it to a dedicated image processing system, obtains the morphological information of the captured target, and converts it into a digital signal according to the pixel distribution, brightness, color, and other information; the image system performs various operations on these signals to extract the features of the target, and then controls the actions of on-site equipment according to the discrimination results.

[0095] Therefore, based on computer vision technology, a machine vision device is used to perform real-time monitoring and continuous acquisition of the product to be detected, and real-time images of the appearance of the product to be detected under different acquisition angles and light intensity conditions can be obtained, providing data support for the detection of product appearance defects.

[0096] Specifically, the acquisition adjustment unit further specifically performs the following steps:

[0097] Extract each light intensity initial value included in the light intensity initial value group for the product;

[0098] After the test product is fixed, take pictures of the test product according to each light intensity initial value to obtain product test images corresponding to each light intensity initial value;

[0099] Compare the gray values of the pixel points included in the product test images corresponding to each light intensity initial value with a preset gray threshold to obtain the area of the target region formed by the pixel points whose gray values exceed the preset gray threshold;

[0100] Obtain a light intensity adjustment coefficient by using the area of the target region of the product images corresponding to each light intensity initial value;

[0101] Among them, the light intensity adjustment coefficient is obtained through the following formula:

[0102]

[0103] Among them, L represents the light intensity adjustment coefficient; n represents the number of light intensity initial values included in the light intensity initial value group; B i represents the i-th light intensity initial value included in the light intensity initial value group; B i+1 represents the (i + 1)-th light intensity initial value included in the light intensity initial value group; S i represents the area of the target region of the product test image corresponding to the i-th light intensity initial value included in the light intensity initial value group; S i+1 represents the area of the target region of the product test image corresponding to the (i + 1)-th light intensity initial value included in the light intensity initial value group; B b represents the standard deviation of the n light intensity initial values included in the light intensity initial value group; S b represents the standard deviation of the areas of the target regions of the product test images corresponding to the n light intensity initial values included in the light intensity initial value group;

[0104] Adjust the light intensity of the image acquisition of the product to be detected by using the light intensity adjustment coefficient.

[0105] The technical effects of the above technical solution are as follows: By setting a group of initial light intensities and taking pictures of the product under each intensity, product images under different lighting conditions can be obtained. This method can systematically explore the influence of light intensity on the quality of product images. By comparing the gray values of pixel points with a preset gray threshold, the size of specific areas (such as over-bright or over-dark areas) in the product image under different lighting conditions can be determined, which provides key information for subsequent light intensity adjustment. The light intensity adjustment coefficient (L) is calculated through a formula, which comprehensively considers the initial light intensity and the change in the area of the target region in the product test image. This method avoids the subjectivity of manual judgment and improves the accuracy and repeatability of light adjustment. The formula includes the standard deviation (Bb) of the initial light intensity and the standard deviation (Sb) of the target region area, which helps to reflect the fluctuations in light intensity and image quality changes, thereby adjusting the light intensity more precisely. Using the calculated light intensity adjustment coefficient, the light intensity for image acquisition of the product to be detected can be adjusted. This helps to ensure that the quality of the product images collected under different lighting conditions remains consistent, thereby improving the accuracy of image analysis, product detection, or quality control. Through automated and precise light intensity adjustment, this technical solution reduces image quality problems and detection errors caused by improper lighting, thereby improving production efficiency and product quality. In addition, this technology can also be applied to various occasions that require image acquisition and processing, such as machine vision, quality inspection, product classification, etc., and has broad application prospects.

[0106] In summary, through precise light intensity adjustment and automated calculation of the light intensity adjustment coefficient, this technical solution optimizes the quality of image acquisition, improves production efficiency and product quality, and has significant technical effects and practical application value.

[0107] Specifically, the adjustment of the light intensity for image acquisition of the product to be detected by the light intensity adjustment coefficient includes:

[0108] Retrieving the initial light intensity value corresponding to the minimum area of the target region in the product test image as the first light intensity;

[0109] Retrieving the initial light intensity value corresponding to the maximum area of the target region in the product test image as the second light intensity;

[0110] Obtaining a light intensity reference value using the first light intensity and the second light intensity;

[0111] Among them, the light intensity reference value is obtained through the following formula:

[0112]

[0113] Among them, B cRepresents the reference value of the light intensity; B p Represents the average value of the initial light intensity values included in the initial light intensity group; B min Represents the first light intensity; B max Represents the second light intensity; n represents the number of the initial light intensity values included in the initial light intensity group; H zi Represents the gray intermediate value corresponding to the target area area corresponding to the i-th initial light intensity value; H ki Represents the central gray value of the product test image corresponding to the i-th initial light intensity value;

[0114] Retrieve the light intensity adjustment coefficient;

[0115] Obtain the light intensity of the product to be detected by using the light intensity adjustment coefficient and the reference value of the light intensity;

[0116] Wherein, the light intensity is obtained by the following formula:

[0117]

[0118] Wherein, B t Represents the light intensity; B c Represents the reference value of the light intensity; L represents the light intensity adjustment coefficient.

[0119] The technical effects of the above technical solution are as follows: By retrieving the initial light intensity values (i.e., the first light intensity and the second light intensity) corresponding to the minimum and maximum target area sizes in the product test image, the light conditions that cause extreme changes in image quality can be identified. This provides an important reference for further determining the light intensity reference value. The calculation formula of the light intensity reference value comprehensively considers the average value of the initial light intensity (Bp), the first light intensity (Bmin), and the second light intensity (Bmax), as well as the central gray value of the target area corresponding to each initial light intensity (Hz i. Note that the definition of Hz i here seems a bit confusing in the original text because usually the central gray value does not change with the initial light intensity. Here it may refer to a gray feature value related to the light intensity). This comprehensive calculation method helps to obtain a more scientific and reasonable light intensity reference value, providing a reliable basis for subsequent adjustment of the light intensity of the product to be detected. By retrieving the light intensity adjustment coefficient calculated previously, it can be combined with the light intensity reference value to further calculate the optimal light intensity (Bt) of the product to be detected. The light intensity adjustment coefficient reflects the degree of influence of light intensity change on the product image quality, so its utilization helps to achieve more accurate light intensity adjustment. Precise adjustment of the light intensity can ensure the image acquisition quality of the product to be detected, reducing problems such as image blurring, overexposure, or underexposure caused by improper lighting. This helps to improve the accuracy of product detection and reduce the risks of false detection and missed detection. Through the automated and intelligent light intensity adjustment process, the number of manual interventions and repeated adjustments can be reduced, thereby optimizing the production process and improving production efficiency. This technical solution is not only applicable to the field of product detection but can also be extended to multiple fields such as machine vision, image processing, and image recognition, providing precise and efficient light intensity adjustment methods for these fields.

[0120] In summary, through precise calculation of the light intensity reference value and effective utilization of the light intensity adjustment coefficient, this technical solution achieves precise adjustment of the light intensity of the product to be detected, improves the image acquisition quality and product detection accuracy, optimizes the production process and efficiency, and has broad application prospects and significant technical effects.

[0121] In this embodiment, as a preferred technical solution of the present invention, the image processing module includes:

[0122] An image denoising unit for denoising the real-time image of the product appearance based on computer vision;

[0123] Obtain the real-time image of the product appearance based on computer vision, convert the real-time image of the product appearance based on computer vision into a grayscale image, and perform denoising processing on the real-time image of the product appearance based on computer vision using a median filter;

[0124] Among them, the window size of the median filter is set according to the noise type and intensity, as well as the image details and size, in the real-time image of the product appearance based on computer vision.

[0125] When the noise in the real-time image of the product appearance based on computer vision is salt-and-pepper noise or random noise and the noise intensity is high, a large window is used; when the noise intensity is low, a small window is used.

[0126] When the image contains important detail or edge information, a small window is used; when the image size is large, a small window is used or a block processing method is adopted.

[0127] After the window size of the median filter is set, for each pixel point in the real-time image of the product appearance based on computer vision, the median of the gray values of all pixel points within the neighborhood of the window size around it is calculated, and the gray value of this pixel point is replaced with the median, which is used to remove or reduce the noise in the real-time image of the product appearance based on computer vision, and the denoised real-time image of the product appearance is saved or displayed.

[0128] In this embodiment, the median filter can adopt a 3*3 window, calculate the median of the gray values of all pixel points within the 3*3 window neighborhood around it, and replace the gray value of this pixel point with the median, which is used to remove or reduce the noise in the real-time image of the product appearance based on computer vision.

[0129] In this embodiment, as a preferred technical solution of the present invention, the image processing module further includes:

[0130] An image adjustment unit, which is used to adjust the brightness and contrast of the denoised real-time image of the product appearance.

[0131] Brightness adjustment: Obtain the gray values of all pixels in the denoised real-time image of the product appearance, increase the brightness by increasing the gray values of all pixels in the denoised real-time image of the product appearance, and reduce the brightness by reducing the gray values of all pixels in the denoised real-time image of the product appearance.

[0132] Contrast adjustment: Based on histogram equalization, the contrast of the denoised real-time image of the product appearance is adjusted. By redistributing the gray values of the real-time image of the product appearance, the gray distribution of the real-time image of the product appearance is adjusted to a uniform distribution, and the dynamic range of the pixel gray values is increased to adjust the contrast of the real-time image of the product appearance.

[0133] In this embodiment, as a preferred technical solution of the present invention, the image semantic module includes:

[0134] An extraction and matching unit, which is used to perform feature extraction and matching on the processed real-time image of the product appearance.

[0135] Feature extraction is performed on the processed real-time product appearance image based on a deep learning model of a convolutional neural network, automatically learning the features of the real-time product appearance image, and extracting highly discriminative feature vectors, including shape features, texture features, and color features. A similarity algorithm is used to match the extracted feature vectors with the features in the existing feature library;

[0136] A semantic segmentation unit is used to perform semantic segmentation on the real-time product appearance image, dividing the real-time product appearance image into different semantic regions, each semantic region having the same class label. The extracted feature vectors and the semantic segmentation results are fused, and the feature vectors are concatenated with the semantic labels to generate a unified semantic image of the product appearance.

[0137] It should be noted that semantic segmentation is an image processing technology aimed at classifying each pixel in the real-time product appearance image into a specific semantic category. It does not distinguish different individuals within the same category but focuses on the semantic content to which the pixel belongs.

[0138] In this embodiment, as a preferred technical solution of the present invention, the defect detection module includes:

[0139] A model training unit is used to train a product appearance defect detection model based on image semantics;

[0140] According to the product appearance defect detection requirements based on image semantics, historical product appearance images are collected;

[0141] The collected historical product appearance images are divided to determine the training set and the test set;

[0142] Based on the training set, the machine learning model is trained and iterated, enabling the machine learning model to automatically learn product appearance defect detection and determining a product appearance defect detection model based on image semantics;

[0143] Based on the test set, the performance of the product appearance defect detection model based on image semantics is tested to determine whether the product appearance defect detection model based on image semantics can achieve the expected effect and determine the optimal product appearance defect detection model based on image semantics.

[0144] In this embodiment, as a preferred technical solution of the present invention, the defect detection module further includes:

[0145] A defect detection unit is used to detect defects in the product appearance;

[0146] Obtain the optimal product appearance defect detection model based on image semantics;

[0147] Deploy the optimal product appearance defect detection model based on image semantics to the actual product appearance defect detection environment;

[0148] Perform image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics, check whether there are defects in the product appearance, and determine the product appearance defect detection result based on image semantics;

[0149] An early warning display unit for giving early warning alarms and visual display of the product appearance defect situation;

[0150] Obtain the product appearance defect detection result based on image semantics;

[0151] When it is detected that there are defects in the product appearance, automatically give an early warning alarm and notify the management personnel in time;

[0152] Based on the product information, mark the product appearance defect situation to form a product appearance defect detection report, and visually display the product appearance defect detection report to the management personnel in a visual form.

[0153] Embodiment 2

[0154] In order to better show the product appearance defect detection process based on image semantics, this embodiment now provides a product appearance defect detection method based on image semantics, which is implemented based on the above-mentioned product appearance defect detection system based on image semantics, and includes the following steps:

[0155] S1: Place the product to be detected on the product appearance defect detection station, and based on computer vision technology, perform real-time acquisition of the product to be detected under different acquisition angles and light intensity conditions to determine the real-time product appearance image based on computer vision;

[0156] S2: Based on the median filter, perform denoising processing on the real-time product appearance image based on computer vision, remove or reduce the noise in the real-time product appearance image based on computer vision, and save or display the denoised real-time product appearance image;

[0157] S3: Adjust the brightness of the real-time product appearance image by increasing or decreasing the gray value of all pixels in the real-time product appearance image, and based on histogram equalization, increase the dynamic range of the pixel gray value in the real-time product appearance image to adjust the contrast of the real-time product appearance image;

[0158] S4: Based on the convolutional neural network deep learning model, extract features from the real-time product appearance image, extract highly discriminative feature vectors, and use a similarity algorithm to match the extracted feature vectors with the features in the existing feature library, and perform semantic segmentation on the real-time product appearance image, and fuse the extracted feature vectors and semantic segmentation results to generate a product appearance semantic image:

[0159] S5: Perform image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics, determine whether there are defects in the product appearance. When defects in the product appearance are detected, an early warning alarm is automatically issued to notify the management personnel in a timely manner. Moreover, based on the product information, mark the product appearance defect situation to form a product appearance defect detection report, and visually display the product appearance defect detection report to the management personnel in a visual form, facilitating the management personnel to manage the products with defective appearances in a timely manner.

[0160] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.

[0161] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. The product appearance defect detection system based on image semantics is characterized by: include: Image acquisition module, used to acquire real-time images of product appearance based on computer vision; An image processing module is used to process real-time images of product appearance based on computer vision; Image semantics module, used to extract features and perform semantic segmentation on real-time images of product appearance to generate semantic images of product appearance; The defect detection module is used to detect defects in product appearance based on machine learning technology, determine whether there are defects in the product appearance, and issue early warning alarms and visual displays of product appearance defects.

2. The product appearance defect detection system based on image semantics according to claim 1, characterized in that: The image acquisition module comprises: Product fixing unit, used to fix the product to be tested; Place the product to be tested on the product appearance defect detection station and fix the product to be tested so that it is placed firmly; The acquisition adjustment unit is used to adjust the acquisition angle and light intensity of the product to be inspected; When collecting images of products to be inspected, the collection angle and light intensity of the products to be inspected are adjusted based on the intelligent adjustment device; Product acquisition unit, used to acquire real-time images of product appearance; Based on computer vision technology, machine vision equipment is used to monitor and continuously collect images of the products to be inspected in real time, obtain product images of the products to be inspected under different acquisition angles and light intensity conditions, and determine the real-time image of the product appearance based on computer vision.

3. The product appearance defect detection system based on image semantics according to claim 2, characterized in that: The acquisition adjustment unit further specifically performs the following steps: Extract each initial illumination intensity value contained in the initial illumination intensity group for the product; After the test product is fixed, photograph the test product according to each of the initial light intensity values ​​to obtain a product test image corresponding to each of the initial light intensity values; Compare the grayscale values ​​of the pixels contained in the product test image corresponding to each initial illumination intensity value with a preset grayscale threshold, and obtain the target area formed by the pixels whose grayscale values ​​exceed the preset grayscale threshold; Obtaining a light intensity adjustment coefficient using the target area of ​​the product image corresponding to each initial light intensity value; The light intensity adjustment coefficient is obtained by the following formula: Wherein, L represents the light intensity adjustment coefficient; n represents the number of initial light intensity values ​​contained in the initial light intensity group; B i Indicates the i-th initial illumination intensity value contained in the initial illumination intensity group; B i+1 Indicates the i+1th initial illumination intensity value contained in the initial illumination intensity group; S i represents the target area of ​​the product test image corresponding to the i-th initial illumination intensity value contained in the initial illumination intensity group; S i+1 represents the target area of ​​the product test image corresponding to the i+1th initial illumination intensity value contained in the initial illumination intensity group; B b It represents the standard deviation of n initial illumination intensity values ​​contained in the initial illumination intensity group; S b Indicates the standard deviation of the target area of ​​the product test image corresponding to the n initial illumination intensity values ​​contained in the initial illumination intensity group; The light intensity adjustment coefficient is used to adjust the light intensity of the image acquisition of the product to be inspected.

4. The product appearance defect detection system based on image semantics according to claim 3, characterized in that: The light intensity of the image acquisition of the product to be inspected is adjusted by using the light intensity adjustment coefficient, including: Retrieving an initial illumination intensity value corresponding to a minimum area value of a target region of the product test image as a first illumination intensity; Retrieving an initial illumination intensity value corresponding to the maximum value of the target region area of ​​the product test image as the second illumination intensity; Obtaining a light intensity reference value using the first light intensity and the second light intensity; The illumination intensity reference value is obtained by the following formula: Among them, B c Indicates the reference value of light intensity; B p Represents the average value of the initial illumination intensity values ​​contained in the initial illumination intensity group; B min Indicates the first light intensity; B max represents the second illumination intensity; n represents the number of illumination initial intensity values ​​contained in the illumination initial intensity group; H zi H represents the grayscale intermediate value corresponding to the target area area corresponding to the i-th initial illumination intensity value; ki Indicates the central grayscale value of the product test image corresponding to the i-th initial illumination intensity value; Retrieving the light intensity adjustment coefficient; Obtaining the light intensity of the product to be tested by using the light intensity adjustment coefficient and the light intensity reference value; The light intensity is obtained by the following formula: Among them, B t Indicates light intensity; B c Indicates the reference value of light intensity; L indicates the light intensity adjustment coefficient.

5. The product appearance defect detection system based on image semantics according to claim 2, characterized in that: The image processing module comprises: An image denoising unit, used to denoise the real-time image of product appearance based on computer vision; Acquire a real-time image of the product appearance based on computer vision, convert the real-time image of the product appearance based on computer vision into a grayscale image, and perform denoising on the real-time image of the product appearance based on computer vision based on a median filter; wherein the window size of the median filter is set according to the type and intensity of noise in the real-time image of the product appearance based on computer vision, as well as the image details and size; When the noise in the real-time image of the product appearance based on computer vision is salt and pepper noise or random noise, and the noise intensity is high, a large window is used, and when the noise intensity is low, a small window is used; When the image contains important details or edge information, use a small window. When the image size is large, use a small window or adopt a block processing method. After the window size of the median filter is set, for each pixel in the real-time image of product appearance based on computer vision, the median of the grayscale values ​​of all pixels in the neighborhood of the window size around it is calculated, and the grayscale value of the pixel is replaced by the median to remove or reduce the noise in the real-time image of product appearance based on computer vision, and the denoised real-time image of product appearance is saved or displayed.

6. The product appearance defect detection system based on image semantics according to claim 5, characterized in that: The image processing module further includes: An image adjustment unit, used to adjust the brightness and contrast of the real-time image of the product appearance after denoising; Brightness adjustment: obtaining the grayscale values ​​of all pixels in the real-time image of the product appearance after denoising, increasing the brightness by increasing the grayscale values ​​of all pixels in the real-time image of the product appearance after denoising, and reducing the brightness by reducing the grayscale values ​​of all pixels in the real-time image of the product appearance after denoising; Contrast adjustment: The contrast of the denoised real-time image of product appearance is adjusted based on histogram equalization. The contrast of the real-time image of product appearance is adjusted by redistributing the grayscale value of the real-time image of product appearance, adjusting the grayscale distribution of the real-time image of product appearance to a uniform distribution, and increasing the dynamic range of the pixel grayscale value.

7. The product appearance defect detection system based on image semantics according to claim 6, characterized in that: The image semantic module comprises: An extraction and matching unit, used for extracting and matching features of the processed real-time image of the product appearance; Based on the convolutional neural network deep learning model, the processed real-time image of the product appearance is subjected to feature extraction, and the features of the real-time image of the product appearance are automatically learned to extract highly distinguishable feature vectors, including shape features, texture features, and color features. The extracted feature vectors are matched with the features in the existing feature library using a similarity algorithm. The semantic segmentation unit is used to perform semantic segmentation on the real-time image of product appearance, divide the real-time image of product appearance into different semantic areas, each semantic area has the same category label, fuse the extracted feature vector and the semantic segmentation result, splice the feature vector and the semantic label, and generate a unified semantic image of product appearance.

8. The product appearance defect detection system based on image semantics according to claim 7, characterized in that: The defect detection module comprises: Model training unit, used to train product appearance defect detection model based on image semantics; According to the needs of product appearance defect detection based on image semantics, historical images of product appearance are collected; Divide the collected historical images of product appearance to determine the training set and test set; Based on the training set, the machine learning model is trained and iterated so that the machine learning model can automatically learn product appearance defect detection and determine a product appearance defect detection model based on image semantics; Based on the test set, the performance of the product appearance defect detection model based on image semantics is tested to determine whether the product appearance defect detection model based on image semantics can achieve the expected effect and to determine the optimal product appearance defect detection model based on image semantics.

9. The product appearance defect detection system based on image semantics according to claim 8, characterized in that: The defect detection module further includes: A defect detection unit, used to detect defects on the product appearance; Obtain the optimal product appearance defect detection model based on image semantics; Deploy the optimal product appearance defect detection model based on image semantics in the actual product appearance defect detection environment; Perform image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics, check whether there are defects in the product appearance, and determine the product appearance defect detection result based on image semantics; Early warning display unit, used for early warning alarm and visual display of product appearance defects; Obtain product appearance defect detection results based on image semantics; When defects are detected in the product appearance, an early warning alarm is automatically issued to notify the management personnel in time; Based on product information, product appearance defects are marked to form a product appearance defect detection report, and the product appearance defect detection report is intuitively displayed to management personnel in a visual form.

10. A method for detecting product appearance defects based on image semantics, implemented based on the system for detecting product appearance defects based on image semantics as claimed in claim 9, characterized in that: The steps include: S1: Place the product to be inspected on the product appearance defect inspection station, and based on computer vision technology, collect the product to be inspected in real time under different collection angles and light intensity conditions to determine the real-time image of the product appearance based on computer vision; S2: Based on a median filter, denoising is performed on the real-time image of product appearance based on computer vision, so as to remove or reduce noise in the real-time image of product appearance based on computer vision, and the denoised real-time image of product appearance is saved or displayed; S3: adjusting the brightness of the real-time image of the product appearance by increasing or decreasing the grayscale values ​​of all pixels in the real-time image of the product appearance, and increasing the dynamic range of the pixel grayscale values ​​in the real-time image of the product appearance based on histogram equalization, and adjusting the contrast of the real-time image of the product appearance; S4: Based on the convolutional neural network deep learning model, the real-time image of the product appearance is feature extracted to extract a highly distinguishable feature vector, and the extracted feature vector is matched with the features in the existing feature library using a similarity algorithm. The real-time image of the product appearance is semantically segmented, and the extracted feature vector and the semantic segmentation result are fused to generate a semantic image of the product appearance: S5: Perform image semantic recognition on the product appearance semantic image according to the optimal product appearance defect detection model based on image semantics to determine whether there are defects in the product appearance. When defects are detected in the product appearance, an early warning alarm is automatically issued to notify the management personnel in time, and the product appearance defects are marked based on the product information to form a product appearance defect detection report, which is intuitively displayed to the management personnel in a visual form.

Images