Method and system for workpiece surface defect and character recognition based on multi-vision fusion

Abstract

The present invention discloses a method and system for workpiece surface defect and character recognition based on multi-visual fusion. First, the images collected by multiple industrial cameras are feature fused and spliced, the rough positioning of the surface label of the workpiece to be inspected is completed by SIFT feature matching, and the image is enhanced by the Retinex algorithm to further reduce the influence of illumination; then Gaussian filtering is performed to remove image noise, and in the absence of a large number of defect samples, the GANomaly network is used for defect detection; then the filtered image is adaptively thresholded, and the character is positioned by the CTPN network, and the CRNN network is used for character recognition. Specifically, the CRNN network is trained and verified by adding a training data set of incomplete character samples to improve the recognition effect of the method on incomplete characters; finally, the results of the GANomaly network defect detection and the results of the CRNN network character recognition are visualized and output, and the workpiece to be inspected is sorted according to the results. The present invention has the advantages of accurate detection and sensitive response.

Description

Technical Field

[0001] The present invention belongs to the technical field of surface defect and character recognition of reflective workpieces to be inspected, and in particular relates to a method and system for surface defect and character recognition of workpieces based on multi-vision fusion. Background Art

[0002] In industrial production, it is very important to inspect finished products of workpieces to be inspected. Most of the surfaces of workpieces to be inspected have various types of characters to express production information. However, there are many interferences in the actual inspection process. The messy light sources in the workshop cause reflections on the workpieces to be inspected. There are cracks and wear on the surfaces of the workpieces to be inspected. The materials of the workpieces to be inspected vary greatly, including stainless steel, plastic, rubber, etc. The characters of the workpieces to be inspected are engraved in various ways, including inkjet, embossing, embossing, etc. This brings difficulties to the surface defect detection and character recognition of the workpieces to be inspected, especially cylindrical workpieces to be inspected. How to solve the above problems is of great significance to the goal of realizing the intelligent industrial production of surface defects and character recognition of workpieces to be inspected.

[0003] Although some scholars have studied character recognition on the surface of plane workpieces to be inspected, there are few studies on character recognition on the surface of arc-shaped and cylindrical workpieces to be inspected. Most character information on the surface of cylindrical workpieces to be inspected is often manually read and manually input into the computer for control. This method is labor-intensive and inefficient. As for the detection system, single-camera imaging is generally used for imaging, and the workpiece to be inspected needs to be placed in a specific position in advance, otherwise the character information of the workpiece to be inspected will not be fully collected or even collected. As for the recognition method, the traditional method of character recognition of workpieces to be inspected obtains a binary image through threshold segmentation, and then uses the vertical projection method to segment individual characters, and finally recognizes the characters through template matching. This method has a good effect on simple scenes with fixed detection characters, but for complex planes and when the character style is not fixed, this method requires a lot of time to make a template library, and the poor segmentation effect directly leads to character recognition errors. The deep learning method of character recognition of workpieces to be inspected has good robustness and high recognition accuracy, but there is currently no method for training with incomplete character training sets, and the recognition accuracy of incomplete characters is poor. The surface defect detection method of the workpiece to be inspected is mainly based on deep learning, but the current mainstream defect detection method requires a large number of defect samples. It is difficult to collect a large number of defective workpiece samples during the actual production process of the workpiece to be inspected. Summary of the invention

[0004] In order to solve at least one of the above-mentioned technical problems, the present invention proposes a method and system for workpiece surface defect and character recognition based on multi-vision fusion, which can adapt to the material and character engraving type of cylindrical workpieces, and can quickly identify defects and incomplete characters on the surface of the workpiece to be inspected under the interference of reflective conditions, with accurate detection and sensitive response.

[0005] One of the purposes of the present invention is achieved through the following technical solutions: providing a system for workpiece surface defect and character recognition based on multi-visual fusion, comprising an industrial control computer, a stage, a conveyor belt, multiple industrial cameras, multiple color-adjustable light sources and multiple light source controllers, wherein the industrial control computer is connected to each light source controller and each industrial camera respectively; the stage is mounted on the conveyor belt for placing the workpiece to be inspected, and the workpiece to be inspected is a cylindrical workpiece to be inspected; the conveyor belt is used to transport the workpiece to be inspected; the multiple industrial cameras are distributed around the workpiece to be inspected at a first preset angle in pairs, and are used to capture the surface image of the workpiece to be inspected; the multiple color-adjustable light sources are distributed around the workpiece to be inspected in pairs at a second preset angle in pairs, each color-adjustable light source is provided with three channels, corresponding to red light, green light and blue light respectively, and each light source controller is connected to an adjustable color light source for controlling the adjustable color light source; through the light source controller, the intensity of red light, green light and blue light is adjusted on the industrial control computer to obtain the color light source required by the corresponding adjustable color light source.

[0006] As a further improvement, the number of the industrial cameras and the color-adjustable light sources are both three, the three industrial cameras are evenly distributed around the workpiece to be inspected at an interval of 120° between two, and the three color-adjustable light sources are distributed around the workpiece to be inspected at an interval of 120° between two.

[0007] The second object of the present invention is achieved by the following technical solution: providing a method for workpiece surface defect and character recognition based on multi-vision fusion, using the system for surface defect and character recognition of the workpiece to be inspected based on multi-vision fusion as described above for detection, comprising the following steps:

[0008] S1. Place the workpiece to be inspected on the stage, turn on each industrial camera, each color-adjustable light source, each light source controller and the industrial control computer respectively, and adjust the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected;

[0009] S2, collecting images of the workpiece to be inspected through multiple industrial cameras;

[0010] S3, performing image feature fusion and splicing on images collected by multiple industrial cameras to obtain a spliced ​​image;

[0011] S4, scaling and stitching the images, and using SIFT feature matching to complete the rough positioning of the surface label of the workpiece to be inspected;

[0012] S5. Use the Retinex algorithm to enhance the image, further reduce the influence of light on the workpiece to be inspected, and improve the image quality;

[0013] S6. Using Gaussian spatial filtering method to remove image noise, and using GANomaly network to detect surface defects of the workpiece to be inspected, to obtain defect information of the workpiece to be inspected;

[0014] S7, obtaining a binary image through an adaptive threshold segmentation algorithm;

[0015] S8, using the CTPN network to locate characters on the surface of the workpiece to be inspected;

[0016] S9, reacquire the image sample of the workpiece to be inspected, make a training data set, perform CRNN network training and verification, and send the characters located in step S8 to the trained and verified CRNN network for text recognition to obtain the character recognition result of the surface of the workpiece to be inspected;

[0017] S10, visually outputting the defect information of the workpiece to be inspected obtained in step S6 and the character recognition result of the surface of the workpiece to be inspected obtained in step S9 through an industrial control computer, and displaying them in the spliced ​​image obtained in step S3;

[0018] S11. Determine whether the workpiece to be inspected should be sorted according to the defect information of the workpiece to be inspected displayed in the spliced ​​image and the character recognition result. If there are defects and / or the characters are wrong, sort it; otherwise, do not process it.

[0019] As a further improvement, in step S1, the specific method of adjusting the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected is:

[0020] First, according to the appearance color and label color of the workpiece to be inspected, a light source color that is complementary to the workpiece to be inspected is selected to enhance the contrast color, or a light source color that is a color close to the background of the workpiece to be inspected is selected to remove the interference of unnecessary information; secondly, through the light source controller, the intensity of the red, green or blue light of the corresponding adjustable light source is adjusted on the industrial control computer, and each adjustable light source obtains the required light source color, thereby ensuring the imaging quality of the characters on the surface of the workpiece to be inspected from the imaging aspect.

[0021] As a further improvement, the step S3 specifically includes the following steps:

[0022] S31, scaling the image captured by each industrial camera to obtain a final size while maintaining the aspect ratio, wherein the shortest side is 800 pixels, and extracting feature points from each scaled image using a fast robust feature algorithm;

[0023] S32, performing fast approximate nearest neighbor algorithm matching on the extracted feature points;

[0024] S33, establishing a homography matrix between images, selecting a random sampling consensus algorithm to continue screening reliable matching points, and completing image registration;

[0025] S34. Image fusion is achieved based on the multi-band fusion strategy to eliminate splicing cracks and ghosting phenomena.

[0026] As a further improvement, the image enhancement using the Retinex algorithm in step S5 is specifically performed as follows:

[0027] S51. Based on the Retinex algorithm, the original image S(x, y) in the Retinex algorithm is transformed to the logarithmic domain, so as to convert the product relationship into a sum relationship: log S(x, y)=log R(x, y)+log L(x, y), where R(x, y) represents the reflected image of the original image S(x, y), L(x, y) represents the illuminated image, x represents the horizontal coordinate of the image, and y represents the vertical coordinate of the image.

[0028] S52, perform Gaussian convolution on the original image S(x, y) to estimate the illumination image L(x, y), estimate L(x, y) from the original image S(x, y), and remove the illumination image L(x, y) to obtain the reflected image R(x, y), that is:

[0029]

[0030] Where r(x, y) represents the logarithmic domain expression of the reflection image R(x, y), σ represents the Gaussian ambient constant, λ represents the Gaussian distribution constant coefficient, and exp represents the exponential operation with the natural coefficient as the base.

[0031] As a further improvement, the GANomaly network consists of a generator network, a discriminator D, and a reconstruction encoder The generation network includes an encoder GE(x) and a decoder GD(z), and the step S6 of using the GANomaly network to detect surface defects of the workpiece to be inspected specifically includes:

[0032] S61, taking N images of normal workpieces to be inspected, where N is a positive integer greater than zero;

[0033] S62, the obtained image x i The encoder GE(x) is fed to get the latent variable z, which is passed through the decoder GD(z) to get x i The reconstructed image i≤N;

[0034] S63, discriminator D for image x i Judge as true and reconstruct the image It is judged as false, thereby continuously optimizing the reconstructed image With image x i The gap between the ideal reconstructed image and image x i No difference;

[0035] S64, Reconstruction Encoder Reconstructed image Do the encoding again to get the latent variables of the reconstructed image encoding

[0036] S65, the latent variable z obtained by the encoder GE(x) and the reconstruction encoder The latent variables obtained The difference between them is compared with the preset threshold φ. When the difference is greater than the preset threshold φ, it is determined that the image x i is an abnormal sample; otherwise, the image x i is a normal sample, the abnormal sample is a sample with defects on the surface of the workpiece to be inspected, and the normal sample is a sample with no defects on the surface of the workpiece to be inspected.

[0037] As a further improvement, the step S8 of using the CTPN network to locate the characters on the surface of the workpiece to be inspected specifically includes:

[0038] S81, extract features using the VGG16 network, and use the features obtained by the third convolution layer in the fifth convolution block in the VGG16 network as a feature map, the size of which is W×H×C, where W represents width, H represents height, and C represents the predicted category;

[0039] S82. Use a sliding window of size 3*3 to slide on the feature map. Each sliding window can obtain a feature vector of length 3×3×C. The center of each sliding window will predict k offsets relative to the anchor box, where k is a positive integer greater than zero.

[0040] S83, input the features extracted in step S81 into a bidirectional long short-term memory neural network to obtain an output of length W×256, and then connect a 512 fully connected layer to prepare for output;

[0041] S84, the fully connected layer is followed by an output layer, wherein the output layer includes three outputs: 2k vertical coordinates, 2k scores, and k side-refinements, wherein the vertical coordinate represents the height and the y-axis coordinate of the center of each predicted box, the score represents the category information of the k anchor boxes, and the side-refinement represents the horizontal translation of each predicted box;

[0042] S85, using a standard non-maximum suppression algorithm to filter out redundant text prediction areas;

[0043] S86. Use a graph-based text line construction algorithm to merge the obtained text prediction areas into text lines to complete character positioning.

[0044] As a further improvement, the specific method for preparing the training data set in step S9 is:

[0045] S911. Download 1,000 images of the Chinese natural text dataset jointly launched by Tsinghua University and Tencent from Github;

[0046] S912. Download 1000 images of ICDAR_2015 English text dataset from Github.

[0047] S913, the code generates 1000 images of regular Chinese and English complete characters, 2000 images of incomplete characters, and 1000 images of Chinese and English complete characters and 2000 images of incomplete characters together to form a data set;

[0048] S914. Organize the data set. The format of training data in CRNN is LMDB, which saves two types of data, one is picture data and the other is label data. The picture data is a number of pictures with text, and the height of the text accounts for 80% to 90% of the picture height; the label data is in txt text format, and the text content is the text on the picture data, that is, the text name is consistent with the picture name.

[0049] As a further improvement, the trained and verified CRNN network includes a convolution layer, a recurrent layer and a transcription layer. The step S9 of using the trained and verified CRNN network to recognize characters on the surface of the workpiece to be inspected is decomposed into the following process:

[0050] S921, the convolutional layer uses CNN to extract feature sequences from the input image;

[0051] S922, the recurrent layer uses RNN to predict the label distribution of the feature sequence obtained from the convolutional layer;

[0052] S923, the transcription layer uses CTC to convert the label distribution obtained from the circulation layer into the final recognition result through the de-duplication operation.

[0053] The method and system for workpiece surface defect and character recognition based on multi-visual fusion provided by the present invention first perform feature fusion and splicing on images collected by multiple industrial cameras, complete the rough positioning of the surface label of the workpiece to be inspected by SIFT feature matching, and perform image enhancement by Retinex algorithm to further reduce the influence of illumination; then Gaussian filtering is performed to remove image noise, and in the absence of a large number of defect samples, the GANomaly network is used for defect detection; then the filtered image is adaptively thresholded, and the character is positioned by the CTPN network, and the CRNN network is used for character recognition. Specifically, the CRNN network is trained and verified by adding a training data set of incomplete character samples to improve the recognition effect of the method on incomplete characters; finally, the results of the GANomaly network defect detection and the results of the CRNN network character recognition are visualized and output, and the workpiece to be inspected is sorted according to the results. The present invention can adapt to the material and character engraving type of cylindrical workpieces, and can quickly identify the defects and incomplete characters on the surface of the workpiece to be inspected under the interference of reflective conditions, and has the advantages of accurate detection and sensitive response. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0055] Figure 1 It is a structural schematic diagram of an embodiment of a system for workpiece surface defect and character recognition based on multi-vision fusion.

[0056] Figure 2 It is a flow chart of an embodiment of a method for workpiece surface defect and character recognition based on multi-vision fusion.

[0057] Figure 3 This is an image feature fusion and splicing effect diagram of an embodiment of a method for workpiece surface defect and character recognition based on multi-visual fusion.

[0058] Figure 4 This is a rendering of an embodiment of a method for workpiece surface defect and character recognition based on multi-vision fusion. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features therein can be combined with each other without conflict.

[0060] like Figure 1As shown, an embodiment of the present invention provides a system for workpiece surface defect and character recognition based on multi-visual fusion, including an industrial control computer 6, a stage 2, a conveyor belt 4, multiple industrial cameras, multiple color-adjustable light sources and multiple light source controllers 3, the industrial control computer 6 is respectively connected to each light source controller 3 and each industrial camera; the stage 2 is mounted on the conveyor belt 4, and is used to place the workpiece to be inspected, and the workpiece to be inspected is a cylindrical workpiece to be inspected; the conveyor belt 4 is used to transport the workpiece to be inspected; multiple industrial cameras are distributed around the workpiece to be inspected at a first preset angle in pairs, and are used to capture the surface image of the workpiece to be inspected; multiple color-adjustable light sources are distributed around the workpiece to be inspected in pairs at a second preset angle in pairs, each color-adjustable light source is provided with three channels, corresponding to red light, green light, and blue light, respectively, each light source controller 3 is connected to an adjustable color light source, and is used to control the adjustable color light source; through the light source controller 3, the intensity of red light, green light, and blue light is adjusted on the industrial control computer 6 to obtain the color light source required by the corresponding adjustable color light source.

[0061] The present invention aims at the problem that the workshop lighting on the industrial production line is messy and the workpiece reflection is serious, which leads to the surface defects of the workpiece and the difficulty of character recognition. A system for workpiece surface defects and character recognition based on multi-visual fusion is proposed, which includes an industrial camera 1, a color-adjustable light source 2, a light source controller 3, an industrial computer 6, and a conveyor belt 4. Multiple industrial cameras 1 are placed around the workpiece to be inspected at preset angles in pairs to obtain a complete surface image of the workpiece, and multiple color-adjustable light sources 2 are also placed around the workpiece to be inspected at preset angles in pairs. The above system can adjust the intensity of red light, green light, and blue light on the industrial computer 6 through the light source controller 3 according to the specific material of the workpiece to be inspected, the character engraving method, and the appearance color information, to obtain the color light source required by the corresponding color-adjustable light source 2, reduce the influence of uneven lighting from the hardware, and obtain a clear workpiece surface image.

[0062] As a preferred embodiment of the present invention, the number of the above-mentioned industrial cameras 1 and color-adjustable light sources 2 is three, and the three industrial cameras 1 are evenly distributed around the workpiece to be inspected at an interval of 120° between two, and the three color-adjustable light sources 2 are distributed around the workpiece to be inspected at an interval of 120° between two. It should be noted that each color-adjustable light source 2 is preferably arranged between two adjacent industrial cameras 1. Of course, there may be other possible numbers of industrial cameras 1 and color-adjustable light sources 2, all of which can achieve the technical effects of the present invention.

[0063] At the same time, if Figure 2 As shown, an embodiment of the present invention further provides a method for workpiece surface defect and character recognition based on multi-vision fusion, which uses the above-mentioned system for surface defect and character recognition of the workpiece to be inspected based on multi-vision fusion to perform inspection, and includes the following steps:

[0064] S1. Place the workpiece to be inspected on the stage, turn on each industrial camera, each color-adjustable light source, each light source controller and the industrial control computer respectively, and adjust the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected;

[0065] Preferably, in this step, the specific method of adjusting the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected is:

[0066] First, according to the appearance color and label color of the workpiece to be inspected, a light source color complementary to the workpiece to be inspected is selected to enhance the contrast color, or a light source color adjacent to the background of the workpiece to be inspected is selected to remove the interference of unnecessary information; secondly, through the light source controller, the intensity of the red light, green light or blue light of the corresponding adjustable light source is adjusted on the industrial control computer, and each adjustable light source obtains the required light source color, thereby ensuring the imaging quality of the characters on the surface of the workpiece to be inspected from the imaging aspect; the reason why the light source color complementary to the workpiece to be inspected is selected to enhance the contrast color, or the light source color adjacent to the background of the workpiece to be inspected is selected to remove the interference of unnecessary information is that according to the color wheel diagram theory, the contrast between the target object and the background can be enhanced by selecting the light source color complementary to the target object, or the contrast between the target object and the background can be filtered out by selecting the light source color adjacent to the background;

[0067] S2, collecting images of the workpiece to be inspected through multiple industrial cameras;

[0068] S3, performing image feature fusion and splicing on images collected by multiple industrial cameras to obtain a spliced ​​image;

[0069] S4, scaling the spliced ​​images, and using SIFT (Scale-invariant feature transform) feature matching to complete the rough positioning of the surface label of the workpiece to be inspected;

[0070] S5. Use the Retinex (Singal Scale Retinex) algorithm to perform image enhancement, further reduce the influence of light on the workpiece to be inspected, and improve image quality;

[0071] S6. Using Gaussian spatial filtering method to remove image noise, and using GANomaly network (Generative Adversarial Networks for Anomaly Detection) to detect surface defects of the workpiece to be inspected, and obtain defect information of the workpiece to be inspected;

[0072] S7, obtaining a binary image through an adaptive threshold segmentation algorithm;

[0073] S8. Using CTPN network (Connectionist Text Proposal Network) to locate characters on the surface of the workpiece to be inspected;

[0074] S9, reacquire the image sample of the workpiece to be inspected, make a training data set, train and verify the CRNN network (Convolutional Recurrent Neural Network), and send the characters located in step S8 to the trained and verified CRNN network for text recognition to obtain the character recognition result of the surface of the workpiece to be inspected;

[0075] S10, visually outputting the defect information of the workpiece to be inspected obtained in step S6 and the character recognition result of the surface of the workpiece to be inspected obtained in step S9 through an industrial control computer, and displaying them in the spliced ​​image obtained in step S3;

[0076] S11. Determine whether the workpiece to be inspected should be sorted according to the defect information of the workpiece to be inspected displayed in the spliced ​​image and the character recognition result. If there are defects and / or the characters are wrong, sort it; otherwise, do not process it.

[0077] The above method first performs feature fusion and splicing on images collected by multiple cameras, completes the rough positioning of the surface label of the workpiece to be inspected through SIFT feature matching, and performs image enhancement through the Retinex method to further reduce the influence of illumination from the algorithm; then Gaussian filtering is performed to remove image noise, and in the absence of a large number of defect samples, the GANomaly network is used for defect detection; then the filtered image is adaptively thresholded, and the character positioning is performed through the CTPN network, and the character recognition is performed by the CRNN network. Specifically, the CRNN network is trained and verified by adding a training data set of incomplete character samples to improve the recognition effect of the method on incomplete characters; finally, the results of the GANomaly network defect detection and the results of the CRNN network character recognition are visualized and output, and the workpiece to be inspected is sorted according to the results. The present invention can adapt to the material and character engraving type of the cylindrical workpiece, and can quickly identify the defects and incomplete characters on the surface of the workpiece to be inspected under the interference of reflective conditions, with accurate detection and sensitive response.

[0078] At the same time, if Figure 3 As shown, step S3 specifically includes the following steps:

[0079] S31, scaling the image captured by each industrial camera to obtain a final size while maintaining the aspect ratio, wherein the shortest side is 800 pixels, and extracting feature points from each scaled image using a Speed ​​Up Robust Feature (SURF) algorithm;

[0080] S32, matching the extracted feature points using the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm;

[0081] S33, establishing a homography matrix between images, selecting a random sample consensus algorithm (RANSAC) to continue screening reliable matching points, and completing image registration;

[0082] S34. Image fusion is achieved based on a multi-band fusion strategy to eliminate splicing cracks and ghosting phenomena.

[0083] As a preferred embodiment of the present invention, the image enhancement using the Retinex algorithm in step S5 is specifically performed as follows:

[0084] S51. Based on the Retinex algorithm, the original image S(x, y) in the Retinex algorithm is transformed into the logarithmic domain, so as to convert the product relationship into a sum relationship: log S(x, y) = log R(x, y) + log L(x, y), where R(x, y) represents the reflected image of the original image S(x, y), L(x, y) represents the illuminated image, x represents the horizontal coordinate of the image, and y represents the vertical coordinate of the image. It should be noted that the basic assumption of the Retinex algorithm is that the original image S(x, y) is the product of the illuminated image L(x, y) and the reflected image R(x, y), that is, S(x, y) = R(x, y)*L(x, y).

[0085] S52, perform Gaussian convolution on the original image S(x, y) to estimate the illumination image L(x, y), estimate L(x, y) from the original image S(x, y), and remove the illumination image L(x, y) to obtain the reflected image R(x, y), that is:

[0086]

[0087] Where r(x, y) represents the logarithmic domain expression of the reflection image R(x, y), σ represents the Gaussian ambient constant, λ represents the Gaussian distribution constant, and exp represents the exponential operation with the natural constant e as the base.

[0088] In addition, it is worth mentioning that the above GANomaly network consists of a generation network, a discriminator D, and a reconstruction encoder The generation network includes an encoder GE(x) and a decoder GD(z). In step S6, the surface defect detection of the workpiece to be inspected using the GANomaly network specifically includes:

[0089] S61, photographing N images of normal workpieces to be inspected, where N is a positive integer greater than zero; it should be noted that the value of N is preferably 500, but the value of N is not limited thereto, and may also be 600, 700 or more;

[0090] S62, the obtained image x i The encoder GE(x) is fed to get the latent variable z, which is passed through the decoder GD(z) to get x i The reconstructed image i≤N;

[0091] S63, discriminator D for image x i Judge as true and reconstruct the image It is judged as false, thereby continuously optimizing the reconstructed image With image x i The gap between the ideal reconstructed image and image x i No difference;

[0092] S64, Reconstruction Encoder Reconstructed image Do the encoding again to get the latent variables of the reconstructed image encoding

[0093] S65, the latent variable z obtained by the encoder GE(x) and the reconstruction encoder The latent variables obtained The difference between them is compared with the preset threshold φ. When the difference is greater than the preset threshold φ, it is determined that the image x i is an abnormal sample; otherwise, the image x i is a normal sample, the abnormal sample is a sample with defects on the surface of the workpiece to be inspected, and the normal sample is a sample with no defects on the surface of the workpiece to be inspected.

[0094] It should be noted that in step S65, the encoder GE(x), the decoder GD(z) and the reconstruction encoder When an abnormal sample is received, the encoder GE(x) and decoder GD(z) will not be applicable to abnormal samples, and the encoded latent variable z and the latent variable obtained by the reconstructed encoder will be The gap is large, that is, the latent variable z obtained by the encoder GE(x) and the reconstruction encoder The latent variables obtained The difference between them is greater than the preset threshold φ.

[0095] As a preferred embodiment of the present invention, the above-mentioned step S8 using the CTPN network to locate the characters on the surface of the workpiece to be inspected specifically includes:

[0096] S81, extract features using the VGG16 network, and use the features obtained by the third convolution layer (conv5_3) in the fifth convolution block in the VGG16 network as a feature map, the size of which is W×H×C, where W represents width, H represents height, and C represents the predicted category; it should be noted that the VGG16 network refers to the prior art and will not be described in detail here;

[0097] S82. Use a sliding window of size 3*3 to slide on the feature map. Each sliding window can obtain a feature vector of length 3×3×C. The center of each sliding window will predict k offsets relative to the anchor box, where k is a positive integer greater than zero.

[0098] S83, input the features extracted in step S81 into a bidirectional long short-term memory neural network (Bi-directional Long Short-Term Memory, Bi-LSTM), obtain an output of length W×256, and then connect a 512 fully connected layer to prepare for output;

[0099] S84, the fully connected layer is followed by an output layer, wherein the output layer includes three outputs: 2k vertical coordinates, 2k scores, and k side-refinements, wherein the vertical coordinate represents the height of each prediction box (bounding box) and the y-axis coordinate of the center (which can determine the upper and lower boundaries), the score represents the category information of the k anchor boxes (character or not), and the side-refinement represents the horizontal translation of each prediction box (bounding box);

[0100] S85, using a standard non-maximum suppression algorithm to filter out redundant text prediction regions (proposals);

[0101] S86. Use a graph-based text line construction algorithm to merge the obtained text prediction areas into text lines to complete character positioning.

[0102] It should be noted that the bounding box is obtained by iterative neural network and reflects the position coordinates of the target object; the anchor is used as a reference for the calculation of the bounding box during the iteration process. Based on this reference, the prediction box generated by the algorithm only needs to be adjusted based on this anchor box; the proposal refers to the area enclosed by the bounding box.

[0103] Furthermore, the specific method for preparing the training data set in step S9 is:

[0104] S911. Download 1,000 images of the Chinese natural text dataset jointly launched by Tsinghua University and Tencent from Github;

[0105] S912. Download 1000 images of ICDAR_2015 English text dataset from Github.

[0106] S913, the code generates 1000 images of regular Chinese and English complete characters, 2000 images of incomplete characters, and 1000 images of Chinese and English complete characters and 2000 images of incomplete characters together to form a data set;

[0107] S914. Organize the data set. The format of training data in CRNN is LMDB, which saves two types of data, one is picture data and the other is label data. The picture data is a number of pictures with text, and the height of the text accounts for 80% to 90% of the picture height; the label data is in txt text format, and the text content is the text on the picture data, that is, the text name is consistent with the picture name.

[0108] At the same time, the trained and verified CRNN network in the present invention includes a convolution layer, a recurrent layer and a transcription layer. The step S9 uses the trained and verified CRNN network to recognize the characters on the surface of the workpiece to be inspected, which is decomposed into the following process:

[0109] S921, the convolution layer uses CNN (convolutional neural network) to extract feature sequences from the input image;

[0110] S922, the recurrent layer uses RNN (Recurrent Neural Network) to predict the label distribution of the feature sequence obtained from the convolutional layer;

[0111] S923, the transcription layer uses CTC (Connectionist Temporal Classification) to convert the label distribution obtained from the recurrent layer into the final recognition result through a de-duplication operation.

[0112] Figure 4This is a rendering of an embodiment of a method for workpiece surface defect and character recognition based on multi-visual fusion. Figure 4 The workpiece selected is a car USB charger for surface defect and character recognition. The specific effect is shown in the figure. It can be seen that the surface defects and characters can be identified relatively clearly.

[0113] In summary, compared with the prior art, the advantages of the present invention are:

[0114] (1) The present invention uses multi-camera imaging to adjust the color of the adjustable color light source according to the appearance of the workpiece to be inspected and the color of the label, thereby ensuring the imaging quality and reducing the interference of light from the hardware perspective. At the same time, after the multi-camera images are fused and spliced, the Retinex image enhancement algorithm is used to improve the uneven lighting, further reduce the influence of lighting from the algorithm perspective, and solve the reflection problem on the workpiece surface.

[0115] (2) The present invention uses a GANomaly network to detect defects on the surface of the workpiece to be inspected. It only needs normal sample training, not a large number of defect samples, to achieve significant defect detection results. Compared with the prior art, which generally uses manual image editing to produce a large number of defect samples and then conducts neural network training, it saves time and effort, and has accurate detection.

[0116] (3) The character recognition training data set includes 1,000 images of the Chinese natural text data set jointly launched by Tsinghua University and Tencent, 1,000 images of the ICDAR_2015 English text data set, 1,000 images of regular Chinese and English complete characters, and 2,000 images of incomplete characters. The CRNN network is trained and verified using the generated training data set. The trained and verified CRNN network is highly robust and can quickly and accurately recognize Chinese and English characters regardless of the way the characters are engraved. It also has a good recognition effect on incomplete Chinese and English characters.

[0117] Many specific details are set forth in the above description to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein, and therefore, they should not be construed as limiting the scope of protection of the present invention.

[0118] In summary, although the present invention lists the above-mentioned preferred embodiments, it should be noted that although technicians in this field can make various changes and modifications, unless such changes and modifications deviate from the scope of the present invention, they should be included in the protection scope of the present invention.

Claims

1. A system for workpiece surface defect and character recognition based on multi-vision fusion, characterized in that: It includes an industrial control computer, a stage, a conveyor belt, multiple industrial cameras, multiple color-adjustable light sources and multiple light source controllers, wherein the industrial control computer is connected to each light source controller and each industrial camera respectively; the stage is mounted on the conveyor belt and is used to place the workpiece to be inspected, and the workpiece to be inspected is a cylindrical workpiece to be inspected; the conveyor belt is used to transport the workpiece to be inspected; multiple industrial cameras are arranged around the workpiece to be inspected at a first preset angle in pairs, and are used to capture the surface image of the workpiece to be inspected; multiple color-adjustable light sources are arranged around the workpiece to be inspected at a second preset angle in pairs, and each color-adjustable light source is provided with three channels, corresponding to red light, green light and blue light respectively, and each light source controller is connected to an adjustable color light source, and is used to control the adjustable color light source; through the light source controller, the intensity of red light, green light and blue light is adjusted on the industrial control computer to obtain the color light source required by the corresponding adjustable color light source; The system performs detection, comprising the following steps: S1. Place the workpiece to be inspected on the stage, turn on each industrial camera, each color-adjustable light source, each light source controller and the industrial control computer respectively, and adjust the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected; S2, collecting images of the workpiece to be inspected through multiple industrial cameras; S3, performing image feature fusion and splicing on images collected by multiple industrial cameras to obtain a spliced ​​image; S4, scaling and stitching the images, and using SIFT feature matching to complete the rough positioning of the surface label of the workpiece to be inspected; S5. Use the Retinex algorithm to enhance the image, further reduce the influence of light on the workpiece to be inspected, and improve the image quality; S6. Using Gaussian spatial filtering method to remove image noise, and using GANomaly network to detect surface defects of the workpiece to be inspected, to obtain defect information of the workpiece to be inspected; S7, obtaining a binary image through an adaptive threshold segmentation algorithm; S8, using the CTPN network to locate characters on the surface of the workpiece to be inspected; S9, reacquire the image sample of the workpiece to be inspected, make a training data set, perform CRNN network training and verification, and send the characters located in step S8 to the trained and verified CRNN network for text recognition to obtain the character recognition result of the surface of the workpiece to be inspected; S10, visually outputting the defect information of the workpiece to be inspected obtained in step S6 and the character recognition result of the surface of the workpiece to be inspected obtained in step S9 through an industrial control computer, and displaying them in the spliced ​​image obtained in step S3; S11, judging whether the workpiece to be inspected should be sorted according to the defect information of the workpiece to be inspected displayed in the spliced ​​image and the character recognition result, if there is a defect and / or the character is wrong, then it is sorted; otherwise, it is not processed; In step S1, the specific method of adjusting the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected is: First, according to the appearance color and label color of the workpiece to be inspected, select the light source color that is complementary to the workpiece to be inspected to enhance the contrast color, or select the light source color that is adjacent to the background of the workpiece to be inspected to remove the interference of unnecessary information; secondly, through the light source controller, adjust the intensity of the red light, green light or blue light of the corresponding color-adjustable light source on the industrial control computer, so that each color-adjustable light source obtains the required light source color, ensuring the imaging quality of the characters on the surface of the workpiece to be inspected from the imaging aspect; The specific method for preparing the training data set in step S9 is: S911. Download 1,000 images of the Chinese natural text dataset jointly launched by Tsinghua University and Tencent from Github; S912. Download 1000 images of ICDAR_2015 English text dataset from Github. S913, the code generates 1000 images of regular Chinese and English complete characters, 2000 images of incomplete characters, and 1000 images of Chinese and English complete characters and 2000 images of incomplete characters together to form a data set; S914. Arrange the data set. The format of the training data in CRNN is LMDB, which stores two types of data: image data and label data. The image data is a number of images with text, and the text height accounts for 80% to 90% of the image height. The label data is in txt text format, and the text content is the text on the image data, that is, the text name is consistent with the image name. The step S3 specifically includes the following steps: S31, scaling the image captured by each industrial camera to obtain a final size while maintaining the aspect ratio, wherein the shortest side is 800 pixels, and extracting feature points from each scaled image using a fast robust feature algorithm; S32, performing fast approximate nearest neighbor algorithm matching on the extracted feature points; S33, establishing a homography matrix between images, selecting a random sampling consensus algorithm to continue screening reliable matching points, and completing image registration; S34. Image fusion is achieved based on the multi-band fusion strategy to eliminate splicing cracks and ghosting phenomena.

2. The system for workpiece surface defect and character recognition based on multi-vision fusion according to claim 1 is characterized in that: The number of the industrial cameras and the color-adjustable light sources are both three. The three industrial cameras are evenly distributed around the workpiece to be inspected at an interval of 120° between two, and the three color-adjustable light sources are distributed around the workpiece to be inspected at an interval of 120° between two.

3. A method for workpiece surface defect and character recognition based on multi-vision fusion, characterized in that: Detection using the system for workpiece surface defects and character recognition based on multi-vision fusion as described in any one of claims 1 to 2 comprises the following steps: S1. Place the workpiece to be inspected on the stage, turn on each industrial camera, each color-adjustable light source, each light source controller and the industrial control computer respectively, and adjust the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected; S2, collecting images of the workpiece to be inspected through multiple industrial cameras; S3, performing image feature fusion and splicing on images collected by multiple industrial cameras to obtain a spliced ​​image; S4, scaling and stitching the images, and using SIFT feature matching to complete the rough positioning of the surface label of the workpiece to be inspected; S5. Use the Retinex algorithm to enhance the image, further reduce the influence of light on the workpiece to be inspected, and improve the image quality; S6. Using Gaussian spatial filtering method to remove image noise, and using GANomaly network to detect surface defects of the workpiece to be inspected, to obtain defect information of the workpiece to be inspected; S7, obtaining a binary image through an adaptive threshold segmentation algorithm; S8, using the CTPN network to locate characters on the surface of the workpiece to be inspected; S9, reacquire the image sample of the workpiece to be inspected, make a training data set, perform CRNN network training and verification, and send the characters located in step S8 to the trained and verified CRNN network for text recognition to obtain the character recognition result of the surface of the workpiece to be inspected; S10, visually outputting the defect information of the workpiece to be inspected obtained in step S6 and the character recognition result of the surface of the workpiece to be inspected obtained in step S9 through an industrial control computer, and displaying them in the spliced ​​image obtained in step S3; S11, judging whether the workpiece to be inspected should be sorted according to the defect information of the workpiece to be inspected displayed in the spliced ​​image and the character recognition result, if there is a defect and / or the character is wrong, then it is sorted; otherwise, it is not processed; In step S1, the specific method of adjusting the light source color of each corresponding color-adjustable light source through the industrial control computer and each light source controller to ensure the imaging effect of the workpiece to be inspected is: First, according to the appearance color and label color of the workpiece to be inspected, select the light source color that is complementary to the workpiece to be inspected to enhance the contrast color, or select the light source color that is adjacent to the background of the workpiece to be inspected to remove the interference of unnecessary information; secondly, through the light source controller, adjust the intensity of the red light, green light or blue light of the corresponding color-adjustable light source on the industrial control computer, so that each color-adjustable light source obtains the required light source color, ensuring the imaging quality of the characters on the surface of the workpiece to be inspected from the imaging aspect; The specific method for preparing the training data set in step S9 is: S911. Download 1,000 images of the Chinese natural text dataset jointly launched by Tsinghua University and Tencent from Github; S912. Download 1000 images of ICDAR_2015 English text dataset from Github. S913, the code generates 1000 images of regular Chinese and English complete characters, 2000 images of incomplete characters, and 1000 images of Chinese and English complete characters and 2000 images of incomplete characters together to form a data set; S914. Arrange the data set. The format of the training data in CRNN is LMDB, which stores two types of data: image data and label data. The image data is a number of images with text, and the text height accounts for 80% to 90% of the image height. The label data is in txt text format, and the text content is the text on the image data, that is, the text name is consistent with the image name. The step S3 specifically includes the following steps: S31, scaling the image captured by each industrial camera to obtain a final size while maintaining the aspect ratio, wherein the shortest side is 800 pixels, and extracting feature points from each scaled image using a fast robust feature algorithm; S32, performing fast approximate nearest neighbor algorithm matching on the extracted feature points; S33, establishing a homography matrix between images, selecting a random sampling consensus algorithm to continue screening reliable matching points, and completing image registration; S34. Image fusion is achieved based on the multi-band fusion strategy to eliminate splicing cracks and ghosting phenomena.

4. The method for workpiece surface defect and character recognition based on multi-vision fusion according to claim 3 is characterized in that: The image enhancement using the Retinex algorithm in step S5 is specifically performed as follows: S51. Based on the Retinex algorithm, the original image S(x, y) in the Retinex algorithm is transformed into the logarithmic domain, so as to convert the product relationship into a sum relationship: logS(x, y)=logR(x, y)+logL(x, y), where R(x, y) represents the reflected image of the original image S(x, y), L(x, y) represents the illumination image, x represents the horizontal coordinate of the image, and y represents the vertical coordinate of the image; S52, perform Gaussian convolution on the original image S(x,y) to estimate the illumination image L(x,y), estimate L(x,y) from the original image S(x,y), and remove the illumination image L(x,y) to obtain the reflected image R(x,y), that is: Where r(x, y) represents the logarithmic expression of the reflected image R(x, y), σ represents the space constant around the Gaussian, λ represents the Gaussian distribution constant coefficient, and exp represents the exponential operation with the natural constant e as the base.

5. The method for workpiece surface defect and character recognition based on multi-vision fusion according to claim 3, characterized in that: The GANomaly network consists of a generating network, a discriminator D, and a reconstruction encoder The generation network includes an encoder GE(x) and a decoder GD(z), and the step S6 of using the GANomaly network to detect surface defects of the workpiece to be inspected specifically includes: S61, taking N images of normal workpieces to be inspected, where N is a positive integer greater than zero; S62, the obtained image x i The encoder GE(x) is fed to get the latent variable z, which is passed through the decoder GD(z) to get x i The reconstructed image i≤N; S63, discriminator D for image x i Judge as true and reconstruct the image It is judged as false, thus continuously optimizing the reconstructed image With image x i The gap between the ideal reconstructed image and image x i No difference; S64, Reconstruction Encoder Reconstructed image Do the encoding again to get the latent variables of the reconstructed image encoding S65, the latent variable z obtained by the encoder GE(x) and the reconstruction encoder The latent variables obtained The difference between them is compared with the preset threshold φ. When the difference is greater than the preset threshold φ, it is determined that the image x i is an abnormal sample; otherwise, the image x i is a normal sample, the abnormal sample is a sample with defects on the surface of the workpiece to be inspected, and the normal sample is a sample with no defects on the surface of the workpiece to be inspected.

6. The method for workpiece surface defect and character recognition based on multi-vision fusion according to claim 5, characterized in that: The step S8 of using the CTPN network to locate characters on the surface of the workpiece to be inspected specifically includes: S81, extract features using the VGG16 network, and use the features obtained by the third convolution layer in the fifth convolution block in the VGG16 network as a feature map, the size of which is W×H×C, where W represents width, H represents height, and C represents the predicted category; S82. Use a sliding window of size 3*3 to slide on the feature map. Each sliding window can obtain a feature vector of length 3×3×C. The center of each sliding window will predict k offsets relative to the anchor box, where k is a positive integer greater than zero. S83, input the features extracted in step S81 into a bidirectional long short-term memory neural network to obtain an output of length W×256, and then connect a 512 fully connected layer to prepare for output; S84, the fully connected layer is followed by an output layer, wherein the output layer includes three outputs: 2k vertical coordinates, 2k scores, and k side-refinements, wherein the vertical coordinates represent the height and the y-axis coordinates of the center of each predicted box, the scores represent the category information of the k anchor boxes, and the side-refinements represent the horizontal translation of each predicted box; S85, using a standard non-maximum suppression algorithm to filter out redundant text prediction areas; S86. Use a graph-based text line construction algorithm to merge the obtained text prediction areas into text lines to complete character positioning.

7. The method for workpiece surface defect and character recognition based on multi-vision fusion according to claim 6, characterized in that: The trained and verified CRNN network includes a convolution layer, a recurrent layer and a transcription layer. The step S9 uses the trained and verified CRNN network to identify the characters on the surface of the workpiece to be inspected, which is decomposed into the following process: S921, the convolutional layer uses CNN to extract feature sequences from the input image; S922, the recurrent layer uses RNN to predict the label distribution of the feature sequence obtained from the convolutional layer; S923, the transcription layer uses CTC to convert the label distribution obtained from the circulation layer into the final recognition result through the de-duplication operation.

Images