Image adaptive enhancement method and system based on deep learning

Abstract

The invention discloses an image adaptive enhancement method and system based on deep learning, and relates to the field of remote control. The stamping plate image acquisition module is used for acquiring an image of a to-be-detected stamping plate of a production line through an industrial camera; the deep learning-based image adaptive enhancement method and system comprises the following steps: collecting an image of a stamped plate: storing and saving the image of the stamped plate through an industrial camera by the system; according to the step of image processing, due to the fact that noise and uneven illumination exist in collected stamping plate images, image noise reduction and filtering algorithm processing are carried out on the images, and stamping plate surface defect detection is mainly carried out on the input stamping plate images through a defect classification algorithm. And information about whether the surface of the stamped plate has defects and defect positions is stored.

Description

Technical Field

[0001] The present invention relates to remote control technology, and in particular to an image adaptive enhancement method and system based on deep learning. Background Art

[0002] Stamping sheet defect detection is an important part of modern production, and the level of detection technology will directly affect the quality of stamping sheets. At present, most stamping sheets are still inspected by manual inspection, which not only has a large workload, but is also susceptible to subjective factors of quality inspectors, and cannot guarantee the accuracy and reliability of the inspection. The stamping sheet defect detection system based on machine vision has the advantages of fast detection speed, low cost and reliable detection results.

[0003] Machine vision inspection technology is that under certain lighting conditions (infrared, LED light source, etc.), the industrial camera collects the image of the stamping sheet to the edge server, and then uses image enhancement technology and machine vision technology to perform defect detection on the collected stamping sheet.

[0004] There are many types of defects in stamped sheets. Common defects in stamped sheets are mainly white spots, black spots, scratches, bruises and roller marks. These defects are accompanied by color difference of stamped sheets. Industrial cameras are used to obtain images of stamped sheets. Based on the images obtained by the camera, the defects of stamped sheets can be automatically detected using machine vision methods. The detection efficiency can be synchronized with the production rhythm.

[0005] In the currently available technology, the stamping sheet defect detection method is implemented by first using an image enhancement method and then a defect detection method. The defects of the stamping sheet can be generally divided into three types of defects: point type, surface type and linear type. Summary of the invention

[0006] The purpose of the present invention is to provide an image adaptive enhancement method and system based on deep learning to solve the above-mentioned deficiencies in the prior art.

[0007] In order to achieve the above object, the present invention provides the following technical solution: an image adaptive enhancement method and system based on deep learning, comprising: Stamping sheet image acquisition module: used to acquire the image of the stamping sheet to be inspected on the production line through an industrial camera; Stamping sheet image preprocessing module: used to perform image preprocessing based on the image of the stamping sheet on the target production line; Stamping sheet defect assessment module: used to identify the quality influence coefficient of each stamping sheet defect on the production line, compare and screen out qualified stamping sheets, and mark the defect type and location when unqualified stamping sheets are screened out; Stamping sheet data repository: used to store various types of stamping sheets and related parameter weights.

[0008] Furthermore, a plurality of industrial cameras are distributed at the stamping sheet workstations, the cameras are equipped with strip light sources for fill light, and the strip light sources are equipped with diffusion plates to improve the image imaging effect.

[0009] The image adaptive enhancement method and system based on deep learning, the stamping sheet image preprocessing module includes the following working steps: Collect stamping sheet images, and perform image enhancement on the stamping sheet images using spatial domain, frequency domain filtering, contrast enhancement, morphological enhancement, edge enhancement, and illumination correction, and obtain multiple stamping sheet enhanced images with different enhancements; Image enhancement technology can highlight the interesting information of defects in stamping sheets, while removing some unnecessary information and improving the usability of artwork images; The spatial domain filtering in each of the stamping sheet enhanced images is completed by performing neighborhood operations in the image space. Various spatial domain filters are mainly divided into two categories according to their functions: smoothing filters and sharpening filters. The smoothing filter adopts the super-limit pixel smoothing method. Since the image is a planar two-dimensional pixel distribution, each pixel point is represented by (x, y). The absolute value of the difference between f (x, y) and the domain g (x, y) is compared with the selected threshold value, and the final grayscale g' (x, y) of the point (x, y) is determined according to the comparison result. The expression is:

[0010] Selecting the over-limit pixel smoothing method can protect the details and textures of small grayscale differences.

[0011] Furthermore, the stamping sheet defect assessment module adopts three convolution layers, each convolution layer can obtain the output of all previous convolution layers as input, and adjacent convolution layers are connected through convolution layers and pooling layers.

[0012] Furthermore, the stamping sheet defect assessment module specifically includes the following working steps: Import the stamping sheet image to be detected into the machine vision network, and crop the image into a large unified 64*64dpi; perform normalization on the resized stamping sheet image to be detected, and convert the pixel value range of the stamping sheet image to be detected from [0,255] to [0,1]; The features of the entire image are used to predict each bounding box, and all bounding boxes of an image in all defect categories are also predicted at the same time. The machine vision algorithm first divides the entire stamped sheet image into S*S. If the center of the defect happens to fall in a square, then this square is used to predict the defect. Each square predicts B bounding boxes and the confidence score of the bounding box. These represent the accuracy of the defect and the accuracy of the generated box. The output is a tensor of S*S*(5*B+C). Furthermore, if an object falls within a square, the first item is 1, otherwise it is 0. Each bounding box has five parameters to be predicted, x, y, h, w, confidence, (x, y) represents the coordinates of the center point of the box, which is related to the boundary of the square, (h, w) represents the width and height of the box, which is related to the entire image, and confidence represents the IOU of the predicted box and the label box:

[0013] Each square predicts the probability of three defect categories. During testing, the conditional class probability is multiplied by the confidence prediction of a single box, as shown in the following formula. This formula gives the confidence scores of the three defect categories for each box. These scores reflect the possibility of the defect category appearing in a box and the degree of match between the predicted box and the defect:

[0014] The defect analysis module loss function is used to calculate the gap between the predicted results and the actual results. The loss function is used to quantify the gap and thus determine the accuracy of the prediction. The loss function is a sum of five items. The formula is as follows:

[0015] Furthermore, the method further comprises the following steps: The center point of the prediction box causes the loss in the first line of the above formula, where is the control function. The value is 1 in the box where the stamping sheet defect is included in the defect label, otherwise it is 0. The value and defect label use a simple sum of squares error; The width and height of the prediction box, the loss caused by the second line of the above formula, only the grid points to which the real stamping sheet defects belong will cause losses. Here, the square root is removed from the loss function in order to make the prediction box more inclined to adjust the size to a smaller size; The third and fourth rows of the loss function are the confidence C of the prediction box. When the prediction box does not contain a stamping sheet defect, it is 1, and the label of the confidence is 0; if it contains a stamping sheet defect, the label of the confidence is the IOU value of the prediction box and the real defect box. The fifth row shows the probability P of the defect category of the stamped sheet metal. The corresponding category position has a report value of 1 and other positions have a value of 0.

[0016] Compared with the prior art, the image adaptive enhancement method and system based on deep learning provided by the present invention, through the image acquisition of stamped sheet: the system stores and saves the stamped sheet image through an industrial camera, image processing: since the acquired stamped sheet image has noise and uneven lighting, the image is subjected to image denoising and filtering algorithm processing, stamped sheet surface defect detection: the stamped sheet defect detection part mainly uses a defect classification algorithm to perform defect detection on the input stamped sheet image, and saves whether there are defects on the stamped sheet surface and the defect location information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 A schematic diagram of a lighting structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] Embodiment 1: See also Figure 1 , an image adaptive enhancement system based on deep learning, including: Stamping sheet image acquisition module: used to acquire the image of the stamping sheet to be inspected on the production line through an industrial camera; Stamping sheet image preprocessing module: used to perform image preprocessing based on the image of the stamping sheet on the target production line; Stamping sheet defect assessment module: used to identify the quality influence coefficient of each stamping sheet defect on the production line, compare and screen out qualified stamping sheets, and mark the defect type and location when unqualified stamping sheets are screened out; Stamping sheet data repository: used to store various types of stamping sheets and related parameter weights.

[0021] Multiple industrial cameras are distributed at the stamping sheet workstations. The cameras are equipped with strip light sources for fill light, and the strip light sources are equipped with diffusion plates to improve image imaging effects.

[0022] Embodiment 2: See also Figure 1 ,The image adaptive enhancement method based on deep learning, the stamping sheet image preprocessing module, includes the following working steps: Collect stamping sheet images, and perform image enhancement on the stamping sheet images using spatial domain, frequency domain filtering, contrast enhancement, morphological enhancement, edge enhancement, and illumination correction, and obtain multiple stamping sheet enhanced images with different enhancements; Image enhancement technology can highlight the interesting information of defects in stamping sheets, while removing some unnecessary information and improving the usability of artwork images; The spatial domain filtering in each stamping sheet enhanced image is completed by performing neighborhood operations in the image space. Various spatial domain filters are mainly divided into two categories according to their functions: smoothing filters and sharpening filters. The smoothing filter adopts the super-limit pixel smoothing method. Since the image is a two-dimensional pixel distribution, each pixel point is represented by (x, y). The absolute value of the difference between f(x, y) and the domain g(x, y) is compared with the selected threshold. The final grayscale g'(x, y) of the point (x, y) is determined according to the comparison result. Its expression is:

[0023] Selecting the over-limit pixel smoothing method can protect the details and textures of small grayscale differences.

[0024] The stamping sheet defect assessment module uses three convolutional layers. Each convolutional layer can obtain the output of all previous convolutional layers as input, and adjacent convolutional layers are connected through convolutional layers and pooling layers.

[0025] Due to the uneven illumination of the stamping sheet, the presence of ambient light, and the interference of other light, the collected defect images are prone to uneven illumination. The uneven illumination phenomenon will affect the quality of the stamping sheet image and reduce the accuracy of defect detection. Therefore, it is necessary to study the image preprocessing technology to process the image.

[0026] The stamping sheet defect assessment module specifically includes the following working steps: Import the stamping sheet image to be inspected into the machine vision network, and crop the image into a large unified 64*64dpi; perform normalization on the resized stamping sheet image to be inspected, and convert the pixel value range of the stamping sheet image to be inspected from [0,255] to [0,1]; The features of the entire image are used to predict each bounding box, and all bounding boxes of an image in all defect categories are also predicted at the same time. The machine vision algorithm first divides the entire stamping sheet image into S*S. If the center of the defect happens to fall in a square, then this square is used to predict the defect. Each square predicts B bounding boxes and confidence scores of the bounding boxes. These represent the accuracy of the defect and the accuracy of the generated box. The output is a tensor of S*S*(5*B+C). If an object falls in a square, the first item is 1, otherwise it is 0. Each bounding box has five parameters to be predicted, x, y, h, w, confidence, (x, y) represents the coordinates of the center point of the box, which is related to the boundary of the square, (h, w) represents the width and height of the box, which is related to the entire image, and confidence represents the IOU of the predicted box and the label box:

[0027] Each square predicts the probability of three defect categories. During testing, the conditional class probability is multiplied by the confidence prediction of a single box, as shown in the following formula. This formula gives the confidence scores of the three defect categories for each box. These scores reflect the possibility of the defect category appearing in a box and the degree of match between the predicted box and the defect:

[0028] The defect analysis module loss function is used to calculate the gap between the predicted results and the actual results. The loss function is used to quantify the gap and thus determine the accuracy of the prediction. The loss function is a sum of five items. The formula is as follows:

[0029] The following steps are also included: The center point of the prediction box causes the loss in the first line of the above formula, where is the control function. The value is 1 in the box where the stamping sheet defect is included in the defect label, otherwise it is 0. The value and defect label use a simple sum of squares error; The width and height of the prediction box, the loss caused by the second line of the above formula, only the grid points to which the real stamping sheet defects belong will cause losses. Here, the square root is removed from the loss function in order to make the prediction box more inclined to adjust the size to a smaller size; The third and fourth rows of the loss function are the confidence C of the prediction box. When the prediction box does not contain a stamping sheet defect, it is 1, and the label of the confidence is 0; if it contains a stamping sheet defect, the label of the confidence is the IOU value of the prediction box and the real defect box. The probability P of the defect category of the stamped sheet in the fifth row corresponds to the category position, the report value is 1, and the other positions are 0. The above only describes some exemplary embodiments of the present invention by way of illustration. It is undoubted that for ordinary technicians in this field, the described embodiments can be modified in various different ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. The image adaptive enhancement system based on deep learning is characterized by: include: Stamping sheet image acquisition module: used to acquire the image of the stamping sheet to be inspected on the production line through an industrial camera; Stamping sheet image preprocessing module: used to perform image preprocessing based on the image of the stamping sheet on the target production line; Stamping sheet defect assessment module: used to identify the quality influence coefficient of each stamping sheet defect on the production line, compare and screen out qualified stamping sheets, and mark the defect type and location when unqualified stamping sheets are screened out; Stamping sheet data repository: used to store various types of stamping sheets and related parameter weights.

2. The image adaptive enhancement system based on deep learning according to claim 1, characterized in that: Multiple industrial cameras are distributed at the stamping sheet workstations. The cameras are equipped with strip light sources for fill light, and the strip light sources are equipped with diffusion plates to improve image imaging effects.

3. The image adaptive enhancement method based on deep learning is applicable to the image adaptive enhancement system based on deep learning according to any one of claims 1-2, characterized in that: The stamping sheet image preprocessing module includes the following working steps: Collect stamping sheet images, and perform image enhancement on the stamping sheet images using spatial domain, frequency domain filtering, contrast enhancement, morphological enhancement, edge enhancement, and illumination correction, and obtain multiple stamping sheet enhanced images with different enhancements; Image enhancement technology can highlight the interesting information of defects in stamping sheets, while removing some unnecessary information and improving the usability of artwork images; Extracting each of the stamping sheet enhanced images, the spatial domain filtering is completed by performing neighborhood operations in the image space. Various spatial domain filters are mainly divided into two categories: smoothing filters and sharpening filters according to their functions. The smoothing filter uses the over-limit pixel smoothing method. Since the image is a two-dimensional pixel distribution, each pixel is represented by (x, y). The absolute value of the difference between f(x, y) and the domain g(x, y) is compared with the selected threshold. The final grayscale g'(x, y) of the point (x, y) is determined based on the comparison result. The expression is: ； Selecting the over-limit pixel smoothing method can protect the details and textures of small grayscale differences.

4. The image adaptive enhancement method based on deep learning according to claim 3, characterized in that: The stamping sheet defect assessment module adopts three convolution layers, each convolution layer can obtain the output of all previous convolution layers as input, and adjacent convolution layers are connected through convolution layers and pooling layers.

5. The image adaptive enhancement method based on deep learning according to claim 4, characterized in that: The stamping sheet defect assessment module specifically includes the following working steps: Import the stamping sheet image to be detected into the machine vision network, and crop the image into a large unified 64*64dpi; perform normalization on the resized stamping sheet image to be detected, and convert the pixel value range of the stamping sheet image to be detected from [0,255] to [0,1]; The features of the entire image are used to predict each bounding box, and all bounding boxes of an image in all defect categories are also predicted at the same time. The machine vision algorithm first divides the entire stamping sheet image into S*S. If the center of the defect falls exactly in a square, then this square is used to predict the defect. Each square predicts B bounding boxes and the confidence score of the bounding box, which represent the accuracy of the defect and the accuracy of the generated box. The output is a tensor of S*S*(5*B+C).

6. The image adaptive enhancement method based on deep learning according to claim 5, characterized in that: The confidence score is calculated as follows: If an object falls within a square, the first item is 1, otherwise it is 0. Each bounding box has five parameters to be predicted: x, y, h, w, and confidence. (x, y) represents the coordinates of the center point of the box, which is related to the boundary of the square. (h, w) represents the width and height of the box, which is related to the entire image. Confidence represents the IOU of the predicted box and the label box: Each square predicts the probability of three defect categories. During testing, the conditional class probability is multiplied by the confidence prediction of a single box, as shown in the following formula. This formula gives the confidence scores of the three defect categories for each box. These scores reflect the possibility of the defect category appearing in a box and the degree of match between the predicted box and the defect: The defect analysis module loss function is used to calculate the gap between the predicted results and the actual results. The loss function is used to quantify the gap and thus determine the accuracy of the prediction. The loss function is a sum of five items. The formula is as follows: 。 7. The image adaptive enhancement method based on deep learning according to claim 6, characterized in that: The following steps are also included: The center point of the prediction box causes the loss in the first line of the above formula, where is the control function. The value is 1 in the box where the stamping sheet defect is included in the defect label, otherwise it is 0. The value and defect label use a simple sum of squares error; The width and height of the prediction box, the loss caused by the second line of the above formula, only the grid points to which the real stamping sheet defects belong will cause losses. Here, the square root is removed from the loss function in order to make the prediction box more inclined to adjust the size to a smaller size; The third and fourth rows of the loss function are the confidence C of the prediction box. When the prediction box does not contain a stamping sheet defect, it is 1, and the label of the confidence is 0; if it contains a stamping sheet defect, the label of the confidence is the IOU value of the prediction box and the real defect box. The fifth row shows the probability P of the defect category of the stamped sheet metal. The corresponding category position has a report value of 1 and other positions have a value of 0.

Images