A system and method for appearance defect detection based on fast stereo reconstruction

Abstract

The application discloses an appearance defect detection system and method based on rapid three-dimensional reconstruction, and belongs to the technical field of optical three-dimensional reconstruction defect detection. The method improves the traditional photometric stereo method by flexibly constructing a multi-direction light source illumination image acquisition and processing system, accurately positioning a target area, and calibrating and correcting light intensity, rapidly solves target surface three-dimensional information images by using prior information, and improves the contrast of the metal workpiece surface defect image; the normal vector graph and the depth graph of different channels are used to design an image enhancement algorithm based on multiple channels by using the direction sensitivity advantages of the normal vector graph and the depth graph, and finally, a multi-level YOLO detection model based on multiple views is proposed, the detection results of each view are fused and decided, so that high-precision detection of the metal workpiece surface is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of appearance defect detection system and method based on fast stereoscopic reconstruction, belong to optical three-dimensional reconstruction defect detection technical field. BACKGROUND

[0002] Appearance defect detection is a necessary link of metal workpiece production quality automatic detection, however, surface of each metal workpiece may randomly appear various defects in manufacturing process, such as color difference, obvious defect of size larger;Weak defects such as shallow scratch, shallow concave-convex, pinhole and texture not significant. These defects seriously affect the performance quality of metal workpiece, bring huge loss to enterprise etc. Because the demand and consumption of each metal workpiece is huge, the industry pursues high productivity, high performance and safety more and more strongly. Therefore, in the production process of metal workpiece, the requirement of appearance defect detection is higher and higher. This patent takes cylindrical nickel-hydrogen battery as research object, battery main body shell is made of metal material, and various electrolyte is contained in the inside. Battery surface is divided into upper end face (positive electrode), lower end face (negative electrode) and circumferential surface. Among them, end face has frosted particle feeling, circumferential surface has metal luster and is easy to reflect light, and the difference of different models of battery lies in that the profile of upper and lower end faces is different, and the diameter of battery end face and the height of circumferential surface are different. It is a typical metal workpiece of large batch production, and it is difficult to detect surface weak defects due to small size.

[0003] At present, the detection methods of metal surface defects mainly include manual inspection and machine vision detection. Among them, manual inspection needs to carefully adjust the angle for observation for weak defects, and has the disadvantages of long time consumption, high false detection rate and high missed detection rate. The defect detection technology based on machine vision can overcome the above shortcomings due to non-contact, high efficiency of algorithm, high precision and other advantages. For example, Zhang Bo proposes a one-dimensional partition median, Gaussian filter image preprocessing algorithm (see“Zhang Bo. Visual detection method of cylindrical lithium battery shell circumferential surface defect[D]. Hefei: Hefei University of Technology, 2020 ”), which can eliminate image noise while retaining defect information, and set dynamic threshold according to residual curve to segment battery circumferential surface defects. But this metal surface image detection method is limited by the factors that weak defect texture is not significant in two-dimensional image, especially lacks depth information, which leads to low defect detection rate.

[0004] In order to improve the defect detection rate, based on three-dimensional image weak defect detection is a research hotspot in recent years, mainly including laser scanning method, structured light method, photometric stereo method, binocular stereo vision method and the like. Among the above three-dimensional detection methods, the structured light method and the laser scanning method utilize three-dimensional equipment to obtain point cloud data, and have high precision but high requirements on algorithms and large calculation amount. The binocular stereo vision method needs to calibrate and correct the image matching of the camera, is very sensitive to environmental light, and has low applicability. The photometric stereo method utilizes the brightness relationship of multi-directional light images to solve the three-dimensional information of the surface, does not need three-dimensional scanning equipment and image matching, and has fast calculation but insufficient height reconstruction precision, and the utilization rate of various three-dimensional information obtained by reconstruction is low, and problems such as weak defects with small height change and direction sensitivity being easily missed still exist.

[0005] Among the above methods, the structured light method and the laser scanning method have high requirements on algorithms and large calculation amount, and are not suitable for online detection due to high cost and poor real-time performance; the binocular stereo vision method is not suitable for use in actual production due to its strong sensitivity to environmental light; and the photometric stereo method still has the problem of missing weak defects with small height change and direction sensitivity, which may cause safety hazards; therefore, a new solution is given for the problem. SUMMARY

[0006] In order to accurately and quickly detect the surface defects of a metal workpiece, especially weak defects with small height change and direction sensitivity, the application provides a metal workpiece surface defect detection system and method based on improved photometric stereo, flexibly constructs a multi-directional light source illumination image acquisition and processing system, accurately positions a target area, calibrates and corrects light intensity, improves the traditional photometric stereo method, quickly solves and obtains a three-dimensional information image of the surface of the metal workpiece by using prior information, and improves the contrast of the metal workpiece defect image; the direction sensitivity advantages of normal vector images and depth images of different channels are utilized to design an image enhancement algorithm based on multiple channels, and finally a multi-level YOLO detection model based on multiple views is proposed, the detection results of each view are fused and decided, so as to realize high-precision detection of the surface of the metal workpiece, especially weak defects with small height change and direction sensitivity.

[0007] An appearance defect detection system based on rapid stereo reconstruction, the system is a multi-station and multi-workpiece online detection system, comprising a PLC controller, a rotating clamping mechanism, an image acquisition device, an illumination device and a detection device.

[0008] The multiple stations in the shown detection system respectively realize defect detection on the upper end face, the lower end face and the side face of the target, and a dark shielding plate is arranged between each station; wherein, the illumination device of the upper end face and the lower end face detection station adopts a multi-zone annular light source, and the image acquisition device adopts a plane array camera; the illumination device of the side face detection station adopts a linear stripe light source, and the image acquisition device adopts a line scanning camera; when collecting images, the light source sequentially flashes to collect multi-directional light illumination images.

[0009] Optionally, the multi-zone annular light source is a four-zone annular light source, and during detection, the target is clamped and fixed by a rotating clamping mechanism on the upper end face and the lower end face detection station, and multi-directional light illumination images are obtained by cooperating with the sequential flashing of the multi-zone light source.

[0010] The side face detection station is provided with a rotating platform, and during detection, the target is placed on the rotating platform to rotate, and the linear stripe light source is flashed to realize side face image acquisition.

[0011] Optionally, the four-zone annular light source generates four directions of light, and the azimuth angles are about 0°, 90°, 180° and 270° respectively.

[0012] The application also provides a method for appearance defect detection based on fast stereo reconstruction, which is used for defect detection on the surface of a target, and the method is realized based on the above-mentioned system, and the method comprises the following steps:

[0013] Step 1: for the upper end face and the lower end face of the target, images under four-directional light illumination are collected respectively, and image preprocessing is performed; for the side face of the target, a stripe light illumination image is collected, and high light is suppressed by splitting;

[0014] Step 2: according to the light intensity of each single light source calculated in advance and the images collected in step 1, the normal vector N, the reflectivity p and the depth information of the target surface are calculated by using the improved photometric stereo method, so as to obtain the normal vector graph, the reflectivity graph and the depth graph of the target surface;

[0015] Step 3: based on the normal vector graph and the depth graph, the weight coefficients a, b and g of the optimal enhancement effect are determined by using the defect enhancement algorithm of multi-channel image fusion, and a new enhanced color graph is synthesized according to the weight coefficients and the channel values;

[0016] Step 4: the detection of different types of defects is realized by using a YOLOv5s model, and a class state vector set R(L, x, y, w, h) of the defects is output, wherein L is the type of the detected defects, and x, y, w and h are respectively the center point coordinates and width and height of the predicted area; the types of the defects include scratches, pinholes, concave-convex and dirt;

[0017] Step 5: Set the fusion decision, according to the category state vector output by the YOLOv5s model, make a fusion judgment, and obtain the final detection result of each detection station.

[0018] Optionally, the step 2 calculates the normal vector N, reflectivity p and depth information of the target surface by using the improved photometric stereo method, and the step 2 comprises the following steps of:

[0019] The formula 1 is simplified as follows:

[0020]

[0021]

[0022]

[0023] Wherein, Ic is the pixel value after the light intensity correction; Im is the light intensity correction coefficient; i is the included angle between the unit normal vector and the unit light source vector; s is the light source direction unit vector; n is the unit normal vector; θ is the depression angle of the light source irradiation; The azimuth angle is; the normal vector problem is simplified to solve p and q problem, four partition ring light source generates four direction light illumination, the azimuth angle is about 0°, 90°, 180°, 270° respectively, the characteristics that the sine value and the cosine value are far greater than the opposite value at these special angles are utilized, the four azimuth angles are brought into the above formula, and the opposite relationship of the light source position is utilized to eliminate:

[0024] I sum =Ic0+Ic 90 +Ic 180 +Ic 270 (5)

[0025]

[0026] In formula (6), Ip and Iq are obtained by calculating the corresponding element values of the four images after the light intensity correction; the tangent value of the light source depression angle θ can be calculated in advance after the actual acquisition device layout position is determined. Thus, the normal vector N of the target surface is obtained;

[0027] ρ i =||N i ||2 (7)

[0028]

[0029] M·z=v (9)

[0030] Wherein, v is the gradient value calculated by the normal vector; M is a sparse matrix with a size of (2xm, m), and m is the number of pixels; the linear equation set is solved by the least square method, and finally the height value z is obtained;

[0031] The N, p, z values are linearly normalized into the gray value to obtain a normal vector graph, a reflectivity graph, and a depth graph containing three-dimensional information, and the four single-direction light graphs are weighted to obtain a uniform light original graph.

[0032] Optionally, the step 3 comprises:

[0033] Converting the single-channel gray graph into a pseudo-color depth graph;

[0034] Separating the normal vector graph and the pseudo-color depth graph into corresponding BGR component graphs in sequence;

[0035] Analyzing the contrast of different direction-sensitive defects on different channel gray images, performing image enhancement on different combinations of channel images, and finding an optimal channel image combination scheme by cross-validation;

[0036] Extracting three-dimensional depth features from the depth graph and surface curvature features from the normal vector graph and fusing them;

[0037] Extracting and replacing the Z component reflecting the height direction in the normal vector graph into the original pseudo-color graph to obtain a new enhanced color graph.

[0038] Optionally, the YOLOv5s model in the step 4 comprises four parts: an input layer, a backbone layer, a neck layer, and a prediction layer; wherein the input layer is used for pre-processing the image; the backbone layer is used for feature extraction of the image; the neck layer is used for fusing the feature information extracted by the backbone layer and sending it to the prediction layer; the prediction layer is composed of three scale feature maps, which are used for detecting targets of different sizes.

[0039] Optionally, in the step 5, the fusion decision for the upper end surface and the lower end surface is:

[0040] Case1: When the normal vector graph and the fusion graph detect the same number of defects, the upper end surface / lower end surface defect judgment result does not need to be corrected;

[0041] Case2: When the normal vector graph is 0, the defect judgment result is the fusion graph detection result;

[0042] Case3: When the normal vector graph is 1, if the fusion graph is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion graph is 2, 3, or 4, the upper end surface / lower end surface detection result is corrected to “Comprehensive defect”;

[0043] Case4: When the normal vector graph is 2, if the fusion graph is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion graph is 1, 3, or 4, the upper end surface / lower end surface detection result is corrected to “Comprehensive defect”.

[0044] Case5: when the normal vector map is 3, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 2 or 4, the upper end surface / lower end surface detection result is corrected as “Comprehensive defect”;

[0045] Case6: when the normal vector map is 4, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 2 or 3, the upper end surface / lower end surface detection result is corrected as “Comprehensive defect”.

[0046] Optionally, in the step 5, the fusion decision for the side surface is:

[0047] Case1: when the normal vector map and the uniform illumination map detect the same defect number, the target side surface defect judgment result does not need to be corrected;

[0048] Case2: when the normal vector map is 0, the defect judgment result is the detection result of the uniform illumination map;

[0049] Case3: when the normal vector map is 1, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 2, 3 or 4, the side surface detection result is corrected as “Comprehensive defect”;

[0050] Case4: when the normal vector map is 2, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 1, 3 or 4, the side surface detection result is corrected as “Comprehensive defect”;

[0051] Case5: when the normal vector map is 3, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 1, 2 or 4, the side surface detection result is corrected as “Comprehensive defect”;

[0052] Case6: when the normal vector map is 4, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 1, 2 or 3, the side surface detection result is corrected as “Comprehensive defect”.

[0053] The present application has the following advantages:

[0054] By flexibly constructing a multi-direction light source illumination image acquisition processing system, accurately positioning a target region, and calibrating and correcting light intensity, the traditional photometric stereo method is improved, prior information is used to quickly solve and obtain a battery surface three-dimensional information image, and the image contrast of the battery defect is improved. By using the direction sensitivity advantages of the normal vector maps and the depth maps of different channels, a multi-channel based image enhancement algorithm is designed, and finally a multi-level YOLO detection model based on multi-view is proposed, the detection results of each view are fused and decided to realize high-precision detection of the battery surface. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 is an implementation environment schematic diagram of an appearance defect detection system based on fast stereo reconstruction provided by an embodiment of the present application;

[0057] Figure 2 is a flowchart of image acquisition in the appearance defect detection system based on fast stereo reconstruction provided by an embodiment of the present application;

[0058] Figure 3 is a work flowchart of the appearance defect detection system based on fast stereo reconstruction provided by an embodiment of the present application;

[0059] Figure 4 is a flowchart of the appearance defect detection method based on fast stereo reconstruction provided by an embodiment of the present application;

[0060] Figure 5 is a flowchart of battery surface three-dimensional reconstruction in the appearance defect detection method based on fast stereo reconstruction provided by an embodiment of the present application;

[0061] Figure 6 is a depth map of a different intensity defect sample in an embodiment of the present application;

[0062] Figure 7 is a normal vector map of a direction sensitive defect sample in an embodiment of the present application;

[0063] Figure 8 is a multi-channel map and a fusion map in an embodiment of the present application, wherein (a) is a depth map; (b) is a normal vector map; (c) is a normal vector B channel; (d) is a normal vector G channel; (e) is a normal vector R channel; (f) is a pseudo-color depth map B channel; (g) is a pseudo-color depth map G channel; (h) is a pseudo-color depth map R channel; and (i) is a fusion map.

[0064] Figure 9 YOLOv5s model defect detection flowchart in one embodiment of the application.

[0065] Figure 10 Battery surface defect image example in one embodiment of the application, wherein (a) is the original image; (b) is the light 1; (c) is the light 2; (d) is the light 3; (e) is the light 4; (f) is the fusion image; (g) is the normal vector image; (h) is the depth image; the numbers 1 and 2 after the letters represent the corresponding scratch defects; the numbers 3 and 4 represent the corresponding pinhole defects; the numbers 5 and 6 represent the corresponding concave defects.

[0066] Figure 11 Different image defect training index chart in one embodiment of the application, wherein (a) from left to right, the three images correspond to the average precision mean mAP of the fusion image, the normal vector image and the depth image; (b) from left to right, the three images correspond to the precision value precision of the fusion image, the normal vector image and the depth image; (c) from left to right, the three images correspond to the recall rate recall of the fusion image, the normal vector image and the depth image.

[0067] Figure 12 Different station image detection schematic diagram in one embodiment of the application, wherein (a) is the upper end surface; (b) is the lower end surface; (c) is the cylindrical side surface. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0069] Embodiment one:

[0070] The embodiment provides an appearance defect detection method based on rapid stereo reconstruction, which is realized based on a cylindrical battery surface defect detection system, and the system is a multi-station and multi-workpiece online detection system, which comprises a PLC controller, a rotating clamping mechanism, an image acquisition device, an illumination device and a detection device.

[0071] The multiple stations in the detection system realize defect detection on the upper end surface, the lower end surface and the side surface of the battery respectively, and a dark shielding plate is arranged between the stations; wherein the illumination device of the upper end surface and the lower end surface detection station adopts a multi-zone ring light source, and the image acquisition device adopts a plane array camera; the illumination device of the side surface detection station adopts a linear stripe light source, and the image acquisition device adopts a line scanning camera; when the image is acquired, the light source is sequentially flashed to illuminate and acquire multi-directional light images.

[0072] The multi-partition ring light source is a four-partition ring light source, and during detection, the battery is clamped and fixed by a rotating clamping mechanism on the upper end face and lower end face detection stations, and a multi-directional light image is obtained by sequentially frequency flashing the multi-partition light source.

[0073] The side detection station is provided with a rotating platform, and during detection, the battery is placed on the rotating platform and rotated to realize side image acquisition by frequency flashing a linear stripe light source.

[0074] The four-partition ring light source generates four directions of light, and the azimuth angles are about 0°, 90°, 180° and 270°, respectively.

[0075] The method comprises:

[0076] Step 1: For the upper end face and lower end face of the battery, images under four directions of light are respectively acquired, and image preprocessing is performed; for the side face of the battery, a stripe light image is acquired, and high light is suppressed by splitting;

[0077] Step 2: According to the light intensity of each single light source calculated in advance and the image acquired in step 1, the normal vector N, reflectivity p and depth information of the battery surface are calculated by using an improved photometric stereo method, to obtain a normal vector map, a reflectivity map and a depth map of the battery surface;

[0078] Step 3: Based on the normal vector map and the depth map, a multi-channel image fusion defect enhancement algorithm is used to determine the weight coefficients a, b and g of the optimal enhancement effect, and a new enhanced color map is synthesized according to the weight coefficients and the channel values;

[0079] Step 4: A YOLOv5s model is used to realize detection of different types of defects, and a class state vector set R(L, x, y, w, h) of the defects is output, wherein L is the type of the detected defects, and x, y, w and h are respectively the center point coordinates and width and height of the predicted region; the types of the defects include scratches, pinholes, concave-convex and dirt;

[0080] Step 5: A fusion decision is set, a fusion judgment is made according to the class state vector output by the YOLOv5s model, and a final detection result of each detection station is obtained.

[0081] Embodiment Two

[0082] The embodiment provides a appearance defect detection system and a corresponding detection method based on fast stereo reconstruction, referring to Figure 1 and Figure 2The system is a multi-station, multi-workpiece online detection system, comprising a PLC controller, a rotating clamping mechanism, an image acquisition device, an illumination device, and a detection device. The multiple stations in the detection system respectively realize defect detection on the upper end face, lower end face, and side face of the battery. A dark shielding plate is arranged between each station. The illumination device of the upper end face and lower end face detection station is a multi-zone annular light source, and the image acquisition device adopts a plane array camera. The illumination device of the side face detection station is a linear stripe light source, and the image acquisition device adopts a line scanning camera. When acquiring images, the light source sequentially flashes to acquire multi-directional light illumination images.

[0083] As shown in Figure 3 , under the action of the rotating clamping mechanism, the battery sequentially reaches each detection station to obtain corresponding images. When the battery reaches the upper end face and lower end face detection stations, it is clamped and fixed by the rotating clamping mechanism, and multi-directional light illumination images are obtained by sequentially flashing the multi-zone light source. In practical applications, the multi-zone light source can be a four-zone annular light source, or other number of zone light sources, such as six-zone, eight-zone, etc. This embodiment will be described below using a four-zone annular light source as an example. A rotating platform is arranged on the side face detection station. Under the action of the rotating clamping mechanism, the battery is placed on the rotating platform and rotated, and the side face image acquisition is realized by cooperating with the linear stripe light source flashing.

[0084] Specifically, after the battery is in place, the PLC controller triggers the opening of the stripe light source and the line scanning camera, controls the motor to rotate and output a row signal, takes the rising edge of the camera acquisition signal as the start of the image frame, and uses the camera row exposure as the trigger signal to change cyclically, thereby sequentially acquiring one frame of image in four illumination directions to a preset image height.

[0085] As shown in Figure 4 , after the industrial computer as the detection device receives the images collected by each station (including the positive electrode image, negative electrode image, and side face image), it first judges the station and pre-processes the image, including ROI positioning and cropping of the end face image, and splitting and suppressing highlights of the side face image. Second, an improved four-light source photometric stereo method is used to quickly calculate the normal vector, reflectivity, depth, and other three-dimensional data of the battery surface by introducing the light intensity of each single light source into the correction in advance, and intuitively restoring the three-dimensional topography of the battery surface. A multi-channel image fusion enhancement algorithm based on depth map and normal vector map is proposed for weak defects on the battery surface to improve the contrast of weak defects. Finally, the result image is processed using the YOLOv5 network to predict related defects and then make a decision to fuse, and the detection result is output. Then, the PLC controller outputs the battery from the corresponding detection port according to the detection result.

[0086] Specifically, the traditional photometric stereo method assumes that the surface of the object is Lambertian reflective, and the light is parallel light, and the three-dimensional data of the measured object surface is solved by matrix equation. However, these conditions are difficult to achieve in industrial scenarios, and the calculation is relatively time-consuming. Therefore, the application designs an implementable multi-directional light source irradiation model and an improved photometric stereo fast three-dimensional reconstruction method. The prior information optimization method is used to solve the normal vector, and the traditional matrix equation solving is converted into addition and subtraction arithmetic operations, which greatly reduces the time complexity of the algorithm.

[0087] In the traditional Lambertian ideal diffuse reflection model, the light intensity calculation formula of diffuse reflection light is as follows:

[0088] I=EρLN (1)

[0089] Wherein, I is the brightness of the image sensor; E is the main light intensity of the light source; p is the surface reflectivity of the measured object; L=(L x ,L y ,L z ) is the unit direction vector of the light source; N=(N x ,N y ,N z ) T Is the unit normal vector of the surface of the measured object.

[0090] In order to reduce the time-consuming problem of matrix equation solving, formula (1) is simplified as follows:

[0091]

[0092]

[0093]

[0094] Wherein, Ic is the pixel value of the light intensity correction; Im is the light intensity correction coefficient of the light source; i is the included angle of the unit normal vector and the unit light source vector; s is the unit direction vector of the light source; n is the unit normal vector; theta is the depression angle of the light source illumination; Azimuth angle. The annular partition light source designed by the application generates four direction light illumination, and the azimuth angles are about 0°, 90°, 180° and 270° respectively. The four azimuth angles are brought into the above formula (2), and the opposite relationship of the light source position is used to cancel out:

[0095] I sum =Ic0+Ic 90 +Ic 180 +Ic 270 (5)

[0096]

[0097] In formula 6, Ip and Iq can be calculated by the corresponding element values of the four images after light source light intensity correction; the tangent value of the light source depression angle θ can be calculated in advance after the actual acquisition device layout position is determined, and Ip and Iq are calculated by formula (6), and then formula (4) is brought in. That is, the normal vector calculation is converted into addition and subtraction calculation, and after optimization, the whole calculation process is more efficient and fast.

[0098] The normal vector information is visualized and mapped to the RGB color space to obtain a normal vector diagram as shown in formula 7.

[0099]

[0100] Since the near-field light source illumination of the present application is not ideal parallel light irradiation, the light intensity is uneven, and the light intensity coefficient of the light source is not constant. In order to correct the light intensity of the light source and avoid the influence of the highlight in the image on the subsequent calculation, the light intensity of the light source is corrected by using the distance attenuation characteristic of the near-field light source model. Figure 5 (b) shows the reflectivity of 90% of the diffuse white board image, and the light intensity at each pixel is solved by using the distance attenuation characteristic of the near-field light source model.

[0101]

[0102] Wherein, E is the main light intensity of the light source; g represents the coefficient of the light intensity attenuation of the light source with the angle position, which is the inherent property of the light source; α represents the included angle between the light at the irradiation position and the main optical axis (the brightest point); d represents the distance from the light source to the irradiation position. Each parameter can be calculated by the above calibration. The light intensity can be corrected by being brought into formula (2).

[0103] After the surface normal vector is obtained by the rapid photometric stereo algorithm of the present application, the surface reflectivity ρ and height can be further solved. For a point on any three-dimensional surface z=f(x,y). The normal vector of the point is perpendicular to any two vectors (v1, v2) on the surface of the object, so there is:

[0104] ρ i =||N i ||2 (9)

[0105]

[0106] M·z=v (11)

[0107] Wherein, v is the gradient value calculated by the normal vector; M is a sparse matrix with a size of (2xm, m), and m is the number of pixels. The linear equation set is solved by the least square method, and finally the height value is obtained.

[0108] The normal vector diagram, the reflectivity diagram and the depth diagram containing three-dimensional information are obtained by linearly normalizing the above values into the gray value respectively, and the uniform light illumination original diagram is obtained by weighting the four single-direction light illumination diagrams. As shown in formula 8. Figure 5As shown, the contrast of the same defect in the uniform light source image is higher than that in the non-uniform light source image. Figure 5 (d) compared with the three-dimensional reconstruction result Figure 5 (e) and Figure 5 (f) can highlight the direction-sensitive defects and improve the contrast of the weak defects to some extent.

[0109] As Figure 6 and Figure 7 As shown, for the weak defects such as shallow scratches and shallow depressions, the height features are angle-sensitive, and the height feature contrast of the same defect under different angles will be reduced, which will affect the detection accuracy. By using the feature that the human eye is more sensitive to color information, the present application first converts the single-channel grayscale image into a pseudo-color depth map. In order to further improve the contrast of the weak defects in the image, the normal vector map and the pseudo-color depth map are sequentially separated into corresponding BGR component maps, as shown in Figure 8 Each component represents the bending degree of a point on the surface in different directions. As can be seen, for weak defects, Figure 8 the contrast of the R channel information map of the pseudo-color map shown in (h) in the figure is low, and Figure 8 the R channel map in the normal vector map shown in (e) in the figure has a higher contrast of the weak defects in the Z direction.

[0110] By analyzing the contrast of different direction-sensitive defects on different channel images, image enhancement is performed on different combinations of channel images, and a set of optimal channel image combination schemes is found by using cross-validation. Three-dimensional depth features are extracted from the depth map, surface curvature features are extracted from the normal vector map, and the features are fused. Finally, the Z component reflecting the height direction in the normal vector map is extracted and replaced into the original pseudo-color map, and by assigning different weight coefficients as shown in formula 12, the channel values are combined to synthesize a new enhanced color image, i.e. (i) in the figure. The contrast of the weak defects in the image is improved. The defect image enhancement effect is evaluated by the specific defect contrast index SSIM (Structural similarity), and the optimal weight coefficient for enhancement effect is found. Figure 8

[0111]

[0112] Wherein, B, G, R are the B, G, R channels of the color image, fusionmap is the fusion map, pseudocolormap is the pseudo-color map, normalmap is the normal vector map; α, β, γ are weight coefficients;

[0113] ​By analyzing the features of the three-dimensional reconstruction map and the fusion map of different battery images of each station and the characteristics of various defects. For direction-sensitive defects, reflectivity map and normal vector map are used for detection input, weak defects use enhanced fusion map for detection input, and dirt mainly appears on the side, so the original image is selected for detection input. Different defect types of pictures (including normal vector map, reflectivity map, and fusion map) are used to construct multiple models. Taking the end surface image detection as an example, the flowchart of the network realizing defect detection is shown in Figure 9 .

[0114] First, the image to be detected is input into the YOLOv5s model, which divides the image into 7x7 grids, estimates the Bounding Box and the target class for each grid, and outputs the center coordinates, width and height, and confidence information of the prediction box. The corresponding result image sample of the battery workpiece is classified and trained, where the defect class labels to be detected are scratch, hole, sunken, and sully. The trained model file is used to predict the corresponding image online, and the image prediction confidence and the identified area are marked, as shown in Figure 9 . The model can accurately detect the surface defects of the battery. Whether there is a defect in the image is determined by the confidence and the size of the experience threshold.

[0115] Decision fusion is to fuse the result set of different images of the same station after YOLOv5s model prediction according to certain criteria to determine whether there is a defect in the battery at that station. The fusion decision of the upper end surface and the lower end surface of the present application is as follows:

[0116] Case1: When the normal vector map and the fusion map detect the same defect number, the defect judgment result of the upper end surface / lower end surface does not need to be corrected;

[0117] Case2: When the normal vector map is 0, the defect judgment result is the detection result of the fusion map;

[0118] Case3: When the normal vector map is 1, if the fusion map is 0, it means that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 2, 3, or 4, the upper end surface / lower end surface detection result is corrected to "Comprehensive defect";

[0119] Case4: When the normal vector map is 2, if the fusion map is 0, it means that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 3, or 4, the upper end surface / lower end surface detection result is corrected to "Comprehensive defect";

[0120] Case5: When the normal vector map is 3, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 2 or 4, the upper end surface / lower end surface detection result is corrected as "Comprehensive defect";

[0121] Case6: When the normal vector map is 4, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 2 or 3, the upper end surface / lower end surface detection result is corrected as "Comprehensive defect".

[0122] The fusion decision for the side surface is:

[0123] Case1: When the normal vector map and the uniform illumination map detection defect number are the same, the battery side surface defect judgment result does not need to be corrected;

[0124] Case2: When the normal vector map is 0, the defect judgment result is the detection result of the uniform illumination map;

[0125] Case3: When the normal vector map is 1, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 2, 3 or 4, the side surface detection result is corrected as "Comprehensive defect";

[0126] Case4: When the normal vector map is 2, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 1, 3 or 4, the side surface detection result is corrected as "Comprehensive defect";

[0127] Case5: When the normal vector map is 3, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 1, 2 or 4, the side surface detection result is corrected as "Comprehensive defect";

[0128] Case6: When the normal vector map is 4, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 1, 2 or 3, the side surface detection result is corrected as "Comprehensive defect".

[0129] The specific fusion decision of the present application is:

[0130] Lower end surface Case1: When the reflectivity map and the fusion map prediction are both defect-free, it indicates that the battery lower end surface is defect-free;

[0131] Lower end surface Case2: The battery lower end surface has defects.

[0132] Upper end surface Case 1: when the original drawing and reflectivity graph prediction are both defect-free, it indicates that the upper end surface of the battery is defect-free;

[0133] Upper end surface Case 2: the upper end surface of the battery has defects.

[0134] Cylindrical side surface Case 1: when the original drawing and fusion graph prediction are both defect-free, it indicates that the side surface of the battery is defect-free;

[0135] Cylindrical side surface Case 2: the side surface of the battery has defects.

[0136] Finally, the fusion decision of the detection results of the three surface models of each battery is output.

[0137] The beneficial effects of the present application are: in order to improve the efficiency and accuracy of the cylindrical battery surface detection, a battery surface defect detection system based on improved photometric stereo is designed. The end surface is matched with four partition light sources, the cylindrical side surface selects high-brightness linear stripe stroboscopic light source, and multi-thread concurrent type is used to collect multi-directional light illumination images of the whole surface of the battery. The near-field photometric stereo synthesis algorithm is improved, the near-field light intensity is calibrated and corrected, and the original drawing, reflectivity graph, normal vector graph and depth graph are obtained by quickly calculating the multi-directional light illumination images according to the prior information. A defect enhancement algorithm for multi-channel image fusion is proposed, different three-dimensional feature information is extracted, different weight coefficients are fused, the contrast of weak defects is enhanced, and the recognition rate of the subsequent network is improved. A multi-level model fusion training based on YOLOv5s network is used to complete the defect detection online in real time by using the training file. Through batch experiments, it is verified that the method can improve the detection efficiency while ensuring high detection accuracy of weak scratches and concave-convex defects, basically meets the production demand, and can be extended to other related metal product surface defect detection industry.

[0138] The battery surface defect detection process designed according to the present application is based on the self-built data set to test the effectiveness of the algorithm.

[0139] The test environment is: Intel(R) Core(TM) i7-11800H CPU, 32G memory PC, Win10 operating system, development environment is Visual Studio 2019, including external library functions such as OpenCV and Eigen.

[0140] To verify the effectiveness and stability of the weak defect enhancement algorithm of this invention, the following comparative experiment was designed. Taking the image of the lower end face of a battery with an end face diameter of 9.8mm as an example, 300 batteries with normal appearance but with weak defects, unevenness, or shallow scratches on the lower end face were selected and their images were acquired. These images were first preprocessed and synthesized to obtain depth maps and normal vector maps. Then, the multi-channel fusion defect enhancement algorithm was tested on the two images. By comparing the SSIM values ​​of various fused images, the fusion coefficient was finally selected. The same training samples, validation samples, and test samples were divided into three groups with a ratio of 6:3:1. For the same battery, the corresponding depth map, normal vector map, and fusion... Figure 3 All participants were uniformly labeled and trained.

[0141] To verify the detection performance of the YOLOv5s model on various battery images, the model was evaluated using metrics such as precision (P), recall (R), and mean average precision (mAP). The defect detection results for the three images after prediction by the YOLOv5s deep learning network model are as follows: Figure 11 As shown in the comparison, it can be seen that compared to the original image under single uniform illumination, the difference in grayscale variation of defects is minimal. However, the four-directional illumination device designed in this invention improves the saliency of direction-sensitive defect images through multi-directional lighting. It utilizes the brightness and darkness information of multi-illuminated images to quickly reconstruct the three-dimensional morphology of the battery surface in photometric stereoscopic form, thus enhancing the three-dimensional features of defects to a certain extent. Furthermore, the normal vector map and depth map of weak defects are fused into an enhanced map through multi-channel processing, further improving the contrast of weak defects in the image. This enriches the features of weak defects in the grayscale depth map and solves the problem of contrast differences of defects in different directions in the normal vector map, allowing for better input into the subsequent detection model and improving the model's defect detection rate. Finally, the original grayscale depth map, normal vector map, and the enhanced fused image are compared. Figure 3 The training results of each model are as follows Figure 12 As shown in Table 1, the corresponding defect detection training statistics are as follows. From Table 1, it can be seen that compared to direct training and detection using the original depth map and normal vector map, the average precision and recall of the enhanced fusion map in this invention are improved by 20% to 30%, indicating that the performance of this micro-defect enhancement algorithm is good.

[0142] Table 1 Training results for different image defects

[0143]

[0144] In order to verify the effectiveness and accuracy of the YOLOvs5 network model algorithm, the following experiment is designed. Extract the battery overall appearance images collected by the device in five shifts of actual production operation for the following experiment. Among them, the size of the original image of the upper and lower end face is 2448*2048, and the size of the original image of the side face is 1024*450. The images are processed according to the software algorithm process, and the corresponding different result images are input into the corresponding YOLOv5s network model prediction in turn. The effect picture of each station defect detection is shown as follows Figure 12 In the actual battery production process, due to the high requirements of related processes and machine maintenance, the probability of defective products is low. A total of nearly 15000 battery overall surface images collected in five shifts are detected and counted online as shown in Table 2. Among them, the accuracy rate of battery defect detection is in the range of 99.1% to 99.8%, and the recall rate is in the range of 98.7% and 99.2%, which can meet the actual production requirements.

[0145] Table 2 Defect detection result statistics

[0146]

[0147]

[0148] Part of the steps in the embodiments of the present application can be realized by software, and the corresponding software program can be stored in a readable storage medium, such as a CD or a hard disk.

[0149] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for appearance defect detection based on fast stereo reconstruction for defect detection on a target surface, characterized in that, The appearance defect detection system of the method is realized by rapid stereoscopic reconstruction; The system is a multi-station and multi-workpiece online detection system, comprising a PLC controller, a rotating clamping mechanism, an image acquisition device, an illumination device and a detection device; The multiple stations in the detection system respectively realize defect detection on the upper end face, the lower end face and the side face of the target, and a black shielding plate is arranged between each station; wherein the illumination device of the upper end face and the lower end face detection station adopts a multi-zone annular light source, and the image acquisition device adopts a surface array camera; the illumination device of the side face detection station adopts a linear stripe light source, and the image acquisition device adopts a line scanning camera; when collecting images, the light source sequentially flashes to collect multi-directional light illumination images; The multi-zone annular light source is a four-zone annular light source, and during detection, the target is clamped and fixed by the rotating clamping mechanism on the upper end face and the lower end face detection station, and multi-directional light illumination images are obtained by sequentially flashing the multi-zone light source; The side face detection station is provided with a rotating platform, and during detection, the target is placed on the rotating platform and rotates, and the side face image acquisition is realized by cooperating with the linear stripe light source flashing; The method comprises: Step 1: for the upper end face and the lower end face of the target, images under four direction illuminations are collected respectively, and image preprocessing is performed; for the side face of the target, a stripe light illumination image is collected, and high light is suppressed by splitting; Step 2: According to the pre-calculated light intensity of each single light source and the image collected in step 1, the normal vector N, reflectivity and depth information of the target surface are calculated by using the improved photometric stereo method to obtain the normal vector map, reflectivity map and depth map of the target surface. Step 2: According to the pre-calculated light intensity of each single light source and the image collected in step 1, the normal vector N, reflectivity and depth information of the target surface are calculated by using the improved photometric stereo method to obtain the normal vector map, reflectivity map and depth map of the target surface. Step 3: based on the normal vector map and the depth map, a defect enhancement algorithm of multi-channel image fusion is used to determine the weight coefficient of the optimal enhancement effect , and a new enhanced color map is synthesized according to the weight coefficient and the channel values Step 4: Use the YOLOv5s model to detect different types of defects, output the class state vector set of the defects wherein, is the detected defect type, respectively, the center point coordinates and width and height of the predicted region; the defect types include scratches, pinholes, concave-convex, and dirt. Step 5: set fusion decision, make fusion judgment according to the category state vector output by the YOLOv5s model, and obtain the final detection result of each detection station; Step 2 uses an improved photometric stereo method to calculate the normal vector N and reflectivity of the target surface. and depth information, including: In the traditional Lambert ideal diffuse reflection model, the light intensity calculation formula of diffuse reflection light is as formula 1: (1) wherein, I is the luminance of the image sensor; E is the main light intensity of the light source; is the unit directional vector of the light source; Formula 1 is simplified as follows: (2) (3) (4) wherein, is the pixel value corrected by the light intensity of the light source; is the correction coefficient of the light intensity of the light source; is the included angle between the unit normal vector and the unit light source vector; s is the unit vector of the light source direction; n is the unit normal vector; is the depression angle of the light source irradiation; is the azimuth angle; the problem of solving the normal vector is simplified to solving , the problem, , is the gradient of the surface pixel gray scale in and directions, four-direction light illumination is generated by the four-quadrant ring light source, the azimuth angles are about 0 o , 90 o , 180 o , 270 o , the characteristics that the sine value and the cosine value at these special angles are much larger than the other are utilized, the four azimuth angles are brought into the above formula, and the opposite relationship of the light source position is utilized to cancel out: (5) (6) In formula (5), , , , are the pixel values of the light intensity-corrected light source when the azimuth angle of the four-quadrant ring light source is 0 o , 90 o , 180 o , and 270 o , is the sum of the pixel values of the light intensity-corrected light source in the four directions, used for normalization. In equation (6), and The surface gradient is calculated by taking the corresponding element values ​​from four images after light intensity correction. , The median value, thus the surface gradient , The pixel value can be corrected by the light intensity of the light source. , , , and the angle of depression of the light source To represent; the angle of depression of the light source The tangent value is pre-calculated after the actual layout of the acquisition device is determined; thus, the normal vector N of the target surface is obtained. (7) (8) (9) wherein, The gradient value is calculated for the normal vector; is a sparse matrix with size (2xm, m), m is the number of pixels; the linear equation set is solved by the least square method, and finally the height value is obtained; The , , The normal vector map, reflectivity map and depth map containing three-dimensional information are obtained by linearly normalizing the values into the gray values, respectively, and the uniform light original map is obtained by weighting the four single-direction light map coefficients. The step 3 comprises: Convert the single-channel grayscale image into a pseudo-color depth image; Separate the normal vector image and the pseudo-color depth image into corresponding BGR component images in turn; Analyze the contrast of different direction sensitive defects on different channel grayscale images, perform image enhancement on the channel images in different combinations, and find an optimal channel image combination scheme by cross validation; Extract three-dimensional depth features from the depth image, extract surface curvature features from the normal vector image, and fuse them; Extract and replace the Z component reflecting the height direction in the normal vector image into the original pseudo-color image, combine by assigning different weight coefficients as shown in formula 12, fuse the channel values to obtain a new enhanced color image, evaluate the defect image enhancement effect by the specific defect contrast index SSIM, and find the optimal weight coefficient of the enhancement effect; the formula 12 is expressed as: Wherein, B, G, R are B, G, R channels corresponding to the color image, fusionmap is a fusion map, pseudocolormap is a pseudo-color map, and normalmap is a normal vector map; is a weight coefficient; The step 4 comprises inputting the image to be detected into the YOLOv5s model, the model divides the image into 7x7 grids, estimates the Bounding Box of each grid and the target category of each grid, outputs the center coordinates, width and height of the prediction box and the confidence information, classifies and trains the result image sample corresponding to the battery workpiece, wherein the defect category labels to be detected are scratch, pinhole, concave-convex and dirt, uses the trained model file to predict the corresponding image online, marks the image prediction confidence and the identified area, and judges whether there is a defect by comparing the confidence with the experience threshold value; The fusion decision in step 5 is specifically: Lower end surface Case 1: When both reflectivity map and fusion map prediction are defect-free, it indicates that the lower end surface of the battery is defect-free; Lower end surface Case 2: The lower end surface of the battery has defects; Upper end surface Case 1: When both original map and reflectivity map prediction are defect-free, it indicates that the upper end surface of the battery is defect-free; Upper end surface Case 2: The upper end surface of the battery has defects; Cylindrical side surface Case 1: When both original map and fusion map prediction are defect-free, it indicates that the side surface of the battery is defect-free; Cylindrical side surface Case 2: The side surface of the battery has defects.

2. The method of claim 1, wherein, The quadrants of the annular light source produce light in four directions, azimuthally at approximately 0 o , 90 o , 180 o , and 270 o degrees.

3. The method of claim 1, wherein, The YOLOv5s model in the step 4 includes an input layer, a backbone layer, a neck layer and a prediction layer; wherein the input layer is used for pre-processing the image; the backbone layer is used for feature extraction of the image; the neck layer is used for fusing the feature information extracted by the backbone layer and sending to the prediction layer; the prediction layer is composed of three scale feature maps, which are respectively used for detecting targets of different sizes.

4. The method of claim 3, wherein, In the step 5, the fusion decision for the upper end surface and the lower end surface is: Case 1: When the normal vector map and the fusion map detect the same number of defects, the upper end surface / lower end surface defect judgment result does not need to be corrected; Case 2: When the normal vector map is 0, the defect judgment result is the fusion map detection result; Case 3: When the normal vector map is 1, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 2, 3 or 4, the upper end surface / lower end surface detection result is corrected to "Comprehensive defect"; Case 4: When the normal vector map is 2, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 3 or 4, the upper end surface / lower end surface detection result is corrected to "Comprehensive defect"; Case 5: When the normal vector map is 3, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 2 or 4, the upper end surface / lower end surface detection result is corrected to "Comprehensive defect"; Case 6: When the normal vector map is 4, if the fusion map is 0, it indicates that the upper end surface / lower end surface detection result does not need to be corrected; if the fusion map is 1, 2 or 3, the upper end surface / lower end surface detection result is corrected to "Comprehensive defect".

5. The method of claim 4, wherein, In the step 5, the fusion decision for the side surface is: Case 1: When the normal vector map and the uniform illumination map detect the same number of defects, the side surface defect judgment result does not need to be corrected; Case 2: When the normal vector map is 0, the defect judgment result is the uniform illumination map detection result; Case 3: When the normal vector map is 1, if the uniform illumination map is 0, it indicates that the side surface detection result does not need to be corrected; if the uniform illumination map is 2, 3 or 4, the side surface detection result is corrected to "Comprehensive defect"; Case4: When the normal vector diagram is 2, if the uniform illumination diagram is 0, it indicates that the side detection result does not need to be corrected; if the uniform illumination diagram is 1, 3 or 4, the side detection result is corrected as "Comprehensive defect"; Case5: When the normal vector diagram is 3, if the uniform illumination diagram is 0, it indicates that the side detection result does not need to be corrected; if the uniform illumination diagram is 1, 2 or 4, the side detection result is corrected as "Comprehensive defect"; Case6: When the normal vector diagram is 4, if the uniform illumination diagram is 0, it indicates that the side detection result does not need to be corrected; if the uniform illumination diagram is 1, 2 or 3, the side detection result is corrected as "Comprehensive defect".

Images