A metal surface defect detection method, device and storage medium

Abstract

The application discloses a metal surface defect detection method and device and a storage medium, and belongs to the technical field of defect detection. The method comprises the following steps: using a photometric stereo image acquisition device to acquire images of a high-light metal calibration ball under different illumination directions, and acquiring an illumination direction matrix according to the acquired images and a calibration formula; placing a metal to be detected into the photometric stereo image acquisition device, sequentially lighting each light source, and obtaining an object surface image sequence under different illumination directions; fusing a plurality of object surface images in the object surface image sequence to obtain a fused image; calculating a divergence map of the object surface according to the object surface image sequence and the illumination direction matrix; and inputting the fused image and the divergence map into a preset double-branch feature fusion network respectively to extract fused features of the images and output a final defect detection result. The application improves the defect detection capability by improving the contrast between defects and backgrounds and capturing the geometric topographic information of the surface.

Description

Technical Field

[0001] This invention relates to the field of defect detection technology, and in particular to a method, apparatus and storage medium for detecting defects on metal surfaces. Background Technology

[0002] Sheet metal sheets and parts are essential products in modern manufacturing, with significant applications in various fields such as machinery manufacturing, automobile production, and aerospace. However, various surface defects such as scratches, abrasions, dents, and protrusions are inevitably generated during the manufacturing process. These defects not only affect the product's appearance but may also shorten its lifespan and even pose safety threats. Therefore, to prevent defective products from entering the market, it is essential to conduct product appearance inspection before shipment. Traditional manual visual inspection methods are time-consuming, costly, and their effectiveness is affected by subjective factors. Therefore, there is an urgent need for a vision-based automated inspection method.

[0003] Defect detection methods mainly consist of two modules: image acquisition and detection algorithms. Image acquisition is the foundation of the entire detection process, and the quality of the image is a crucial factor in determining the final detection result. If the defect features are highlighted as much as possible during imaging, the difficulty of subsequent detection can be greatly reduced. This largely depends on the lighting conditions. Some types of defects, such as scratches and abrasions, are usually long and thin with obvious directionality. They can only present high contrast under specific lighting directions and angles. However, most current detection methods only use a single light source for illumination. When the defect orientation does not match the lighting direction, the defect cannot stand out well from the background, leading to missed detections. Defects such as pits and protrusions only have depth variations and do not differ significantly from the background in terms of color and reflectivity. Relying solely on RGB image detection also suffers from low detection rates.

[0004] In addition, in terms of detection algorithms, most of the current mainstream methods are based on general deep learning target detection frameworks. Due to the different shapes and sizes of defects, and the fact that some defects are very similar to the background, these models cannot capture defect feature information well, resulting in problems such as low recognition rate and low detection accuracy. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, device and storage medium for detecting metal surface defects based on photometric stereo imaging and dual-branch feature fusion network.

[0006] The technical solution adopted in this invention is:

[0007] A method for detecting defects on a metal surface includes the following steps:

[0008] A photometric stereo image acquisition device is used to acquire images of a high-gloss metal calibration sphere under different illumination directions, and the illumination direction matrix is ​​obtained based on the acquired images and calibration formula; wherein, the photometric stereo image acquisition device includes multiple light sources in different directions;

[0009] The metal to be tested is placed in the photometric stereo image acquisition device, and each light source is lit in sequence to obtain a sequence of object surface images under different lighting directions.

[0010] Multiple object surface images in the object surface image sequence are fused to obtain a fused image;

[0011] Calculate the scatter map of the object surface based on the object surface image sequence and the illumination direction matrix;

[0012] The fused image and scatter plot are respectively input into a pre-defined dual-branch feature fusion network to extract the fused features of the image and output the final defect detection result.

[0013] Furthermore, the photometric stereo image acquisition device includes:

[0014] An industrial camera;

[0015] Four light sources;

[0016] A light source frame is used to mount the light source, and the four light sources are respectively mounted on the crossbeams in four directions of the light source frame; the four light sources form a non-connected aperture.

[0017] A stage, located in the center of the aperture, is used to place the metal to be measured;

[0018] A camera bracket is used to fix the industrial camera, which is located directly above the stage and shoots vertically downwards.

[0019] Furthermore, the expression for the calibration formula is:

[0020] L=2(V·N)NV

[0021] In the formula, N is the normal vector of the center point of the highlight area, V is the reflected light vector, i.e. the camera's viewing direction, and L is the illumination direction vector of a single light source. The illumination direction vectors of all light sources are combined into an illumination direction matrix.

[0022] Further, the fusion of multiple object surface images in the object surface image sequence includes:

[0023] The contrast pyramid fusion algorithm is used to merge multiple images into one image;

[0024] The contrast pyramid fusion algorithm first obtains the Laplacian pyramid of the image, calculated using the following formula:

[0025]

[0026]

[0027]

[0028] In the formula, ω(m,n) is a Gaussian filter, and G l (x,y) is the image of the l-th layer of the Gaussian pyramid. The image is the interpolated and expanded image of the l-th layer, Z is the highest layer of the Laplace pyramid, and L is the image of the l-th layer. l The image of the l-th layer of the Laplacian pyramid is obtained by subtracting Gaussian pyramids of the same pixel level.

[0029] Will Viewed as G l For a background of (x,y), the contrast pyramid is defined as follows:

[0030]

[0031]

[0032]

[0033]

[0034] In the formula, Let x be the value at position x of the l-th contrast pyramid layer of the k-th image. The values ​​at corresponding positions of the contrast pyramid and Gaussian pyramid after fusion are represented respectively. A new contrast pyramid is constructed by taking the value with the largest absolute value at that position from M images, and a new Gaussian pyramid is constructed by taking the average value of the Gaussian pyramids from M images. The fused image is then restored using the new contrast pyramid and the Gaussian pyramid.

[0035] Further, the step of calculating the scatter map of the object surface based on the object surface image sequence and the illumination direction matrix includes:

[0036] Based on the object surface image sequence and illumination direction matrix, the divergence map and curvature map of the object surface are calculated using the photometric stereo method, divergence formula, and curvature formula.

[0037] The calculation formula for the photometric stereo method is as follows:

[0038] I=Lρn

[0039] In the formula, I is the brightness value at a certain location in the image, L is the calculated illumination direction matrix, ρ is the surface reflectivity of the object, and n is the unit normal vector at that location.

[0040] If we consider ρn as the entire N, then the formula for calculating the normal vector is:

[0041] L T I = L T LN

[0042] N=(L T L) -1 L T I

[0043] ρ=||ρn||=||N||

[0044]

[0045] Furthermore, the expression for the divergence formula is:

[0046]

[0047] The expression for the curvature formula is:

[0048]

[0049] In the formula, p is f x Let f represent the gradient of the object's surface along the x-direction, and q be the gradient of f. y , represents the gradient of the surface along the y-direction, Div represents the divergence at various locations of the object, and K g It represents the curvature at various locations of an object.

[0050] Furthermore, the dual-branch feature fusion network is a dual-branch YOLOv5s feature fusion network, which introduces a residual space channel attention module, as shown in the following expression:

[0051] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0052] M s (F)=σ(Conv(Cat[AvgPool(F),MaxPool(F)]))

[0053] F out =F + F·M c (F)+F·M s (F)

[0054] In the formula, M c M sThese represent the channel attention map and spatial attention map, respectively. F is the input feature, σ is the sigmoid activation function, MLP is a shared multilayer perceptron, Cat is the stacking operation, AgvPool and MaxPool represent average pooling and max pooling, respectively, and Conv is the convolution operation. out For output features.

[0055] Furthermore, the dual-branch feature fusion network performs feature fusion on the features extracted from the two branches using the following formula:

[0056] F fus =Conv(Cat(F) div ,F ori ))

[0057] In the formula, F div F is the feature extracted from the branches of the scatter plot. ori Features extracted from RGB image branches are stacked along the channel dimension. Conv is a convolution operation with half the number of channels, and F... fus This is a dual-branch fusion feature.

[0058] Another technical solution adopted in this invention is:

[0059] A metal surface defect detection device, comprising:

[0060] At least one processor;

[0061] At least one memory for storing at least one program;

[0062] When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.

[0063] Another technical solution adopted in this invention is:

[0064] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0065] The beneficial effects of this invention are as follows: This invention obtains a sequence of object surface images by taking pictures of the metal under different lighting directions, and fuses the defect information of the object surface image sequence into a single image to improve the contrast between defects and background; at the same time, it obtains the corresponding scatter map to capture important geometric shape information of the object surface; the fused enhanced image and the scatter image are input together into a dual-branch feature fusion detection network, and the two branches fully extract the effective information of different images and perform feature-level fusion to effectively enhance the defect detection capability. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart of the steps of a defect detection method based on photometric stereo imaging and a dual-branch feature fusion network in an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram showing the relationship between the incident vector, reflection vector, and normal vector at the highlight point of the calibration sphere in this embodiment of the invention;

[0069] Figure 3 This is a schematic diagram of the photometric stereoscopic image acquisition device in an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of a surface defect detection method based on deep learning and photometric stereo in an embodiment of the present invention;

[0071] Figure 5 This is a schematic diagram of the structure of the improved dual-branch feature fusion detection network in an embodiment of the present invention;

[0072] Figure 6 This is a schematic diagram of the enhanced image of the metal surface under different lighting directions and the contrast pyramid fusion in an embodiment of the present invention;

[0073] Figure 7 These are the scatter map and curvature map of the object surface calculated using a photometric stereo algorithm, divergence and curvature formulas in this embodiment of the invention;

[0074] Figure 8 This is a partial detection result diagram of the comparative test in an embodiment of the present invention. Detailed Implementation

[0075] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0076] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0077] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0078] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0079] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0080] Most current vision-based surface defect detection methods use only a single light source from a fixed direction during illumination. However, some scratches and abrasions are only clearly visible under specific lighting directions. Since the direction of scratches and abrasions cannot be predicted in advance, traditional lighting methods cannot guarantee their effective imaging. Furthermore, some defects, such as pits and bumps, are not significantly different from the normal surface background in terms of color and reflectivity, exhibiting low contrast in the image and sometimes being difficult for the human eye to distinguish, resulting in a low detection rate. Due to the diverse morphologies of defects and limitations imposed by lighting conditions and network feature extraction capabilities, the model cannot obtain sufficient effective information from a single image, leading to less than ideal detection results. In summary, current mainstream deep learning-based surface defect detection methods suffer from the following problems:

[0081] (1) Using only RGB images as input, the network can only judge whether there are defects by the color of the object, reflectivity and shadows formed by lighting. Since the geometric shape information of the target is ignored, it is easy to cause false detection and missed detection.

[0082] (2) The photometric stereo method is combined with deep learning, but only the calculated normal vector map is used as input. Since the gray information contained in the RGB image is ignored, the model cannot obtain enough effective information from the normal vector map, and it is easy to miss some defects with small depth changes.

[0083] (3) Simply stacking RGB images and normal vector images as input is essentially using the normal vector image as a new image channel. However, actual tests show that crosstalk between the two different types of images is easy to occur, which weakens each other's effective feature information and fails to achieve the best detection effect.

[0084] To address the aforementioned issues, this invention proposes a defect detection method based on photometric stereo imaging and a dual-branch feature fusion network. The photometric stereo method employs lighting and imaging from different directions, adapting to the imaging needs of scratches and abrasions from various orientations. Next, a contrast pyramid is used to fuse four images into a single image, effectively enhancing the contrast between the defect and the background. Furthermore, the surface normal vector is obtained through the photometric stereo algorithm, and then the surface divergence is calculated to more effectively capture the geometric shape of the surface under test, highlighting pit and protrusion defects. Finally, two branches extract light reflection information from the RGB image and geometric shape information from the divergence map, respectively, solving the problem that existing technologies cannot effectively detect low-contrast defects.

[0085] like Figure 1 and Figure 4 As shown, this embodiment provides a defect detection method based on photometric stereo imaging and a dual-branch feature fusion network, including the following steps:

[0086] S1. Use a photometric stereo image acquisition device to acquire images of the high-gloss metal calibration sphere under different illumination directions, and obtain the illumination direction matrix based on the acquired images and calibration formula. The photometric stereo image acquisition device includes multiple light sources in different directions.

[0087] Calibrate the light source direction. Using a high-gloss metal sphere as a calibration target for the light source direction, a photometric stereo image acquisition device is used to acquire several images of the calibration sphere under different illumination directions. By identifying the positions of the high-brightness points in the images and combining them with the calibration formula, the illumination direction of the light source is calculated.

[0088] The illumination direction vector is determined using the specular metal sphere calibration method. The metal sphere is placed at a suitable position in the field of view, with the upper left endpoint of the image defined as the origin. The coordinates of the metal sphere's center are then (C...). x C y ), and the light sources in each direction are lit in sequence. At this time, a bright area appears on the surface of the metal sphere, and the coordinates of the center point of the bright area are (P). x ,P y), from which the surface normal vector N = (N x N y N z The calculation formula is:

[0089]

[0090]

[0091]

[0092] In the formula, R is the radius of the calibration sphere, the unit normal vector at the highlight point is obtained, and the reflected light vector is the camera's viewing direction V = (0,0,1).

[0093] In this embodiment, the photometric stereo image acquisition device includes four light sources. However, it should be noted that photometric stereo image acquisition devices with other numbers of light sources should fall within the protection scope of this application. See also Figure 3 The photometric stereo image acquisition device includes an industrial camera and four LED light sources (i.e., Figure 3 The system includes four light sources: a central strip light source 1, a strip light source 2, a strip light source 3, and a strip light source 4 (note that only strip light sources are used in this embodiment, but other shapes of light sources should also fall within the scope of protection of this application); an aluminum profile frame for mounting the light sources; a camera bracket for fixing the camera; and a stage. The four light sources are respectively mounted on aluminum profile beams in four directions, with the stage placed in the center and the camera positioned directly above the stage, shooting vertically downwards.

[0094] Specifically, the metal sphere is placed at a suitable position within the field of view of an industrial camera, and four light sources alternately illuminate it from different directions. The camera captures images under the corresponding lighting conditions, identifies the location of the highlight point, and sets the coordinates of the upper left endpoint of the image to (0,0). Then, the coordinates of the center point of the metal sphere are (C... x C y The coordinates of the highlight point are (P). x ,P y According to the formula, the normal vector N at the highlight point can be obtained as N = (N x N y N z ),like Figure 2 As shown, based on the relationship between the incident light, reflected light, and normal vector, the following expression can be obtained:

[0095] L=2(V·N)NV

[0096] The reflected light vector is the camera's viewing direction (0,0,1), and then the incident light direction vector (L) is calculated. x ,L y ,L z), calculate the illumination direction vectors of all light sources in the same way, and combine them into an illumination direction matrix.

[0097] S2. Place the metal to be tested into the photometric stereo image acquisition device, and light up each light source in sequence to obtain a sequence of object surface images under different lighting directions.

[0098] After calibration, keeping the camera and light source positions unchanged, place the metal to be tested onto the stage, and turn on the light source sequentially to obtain a sequence of object surface images under different lighting directions.

[0099] S3. Fuse multiple object surface images from the object surface image sequence to obtain a fused image.

[0100] In this embodiment, after obtaining four images under different lighting conditions, a Gaussian convolution kernel is used to perform convolution operations on each image. Then, alternating rows and columns are deleted and downsampled to obtain the next level of the Gaussian pyramid. Further convolution and downsampling are then performed to form multiple layers of Gaussian pyramids. The lower-level Gaussian pyramids are expanded by interpolation to make their resolution the same as the upper-level pyramids. The difference between the two yields the Laplacian pyramid for each level, as expressed below:

[0101]

[0102]

[0103]

[0104] The interpolated magnified image of the lower pyramid. It can be viewed as the next level of pyramid G. l Given the background, the contrast pyramid can therefore be defined as follows:

[0105]

[0106] That is, the ratio of each layer of the Laplacian pyramid to the Gaussian pyramid, based on which a three-level contrast pyramid is obtained for four images at different scales.

[0107] For each level of the contrast pyramid, the absolute values ​​at the same pixel location in the four images are compared, and the value with the largest absolute value is taken as the contrast pyramid value of the fused image at that location. The Gaussian pyramid is taken as the average of the Gaussian pyramids of the four images. The fused enhanced image is then reconstructed by combining the contrast pyramid and the Gaussian pyramid. The calculation formula is as follows:

[0108]

[0109]

[0110]

[0111] S4. Calculate the scatter map of the object surface based on the object surface image sequence and the illumination direction matrix.

[0112] Based on four images of the metal plate under different lighting conditions, combined with the calibrated lighting direction matrix, the surface normal vector is calculated, and then the surface divergence map and curvature map are further obtained according to the divergence and curvature formulas.

[0113] Specifically, if we align the image sequence consisting of four images with the illumination direction matrix, and consider the object as a Lambertian solid, then according to the classical photometric stereo method, we obtain the following expression.

[0114] I=Lρn

[0115] Where L is the illumination direction vector matrix, ρ is the surface reflectivity of the object (assumed to be constant), and I is the brightness at a certain location in the image.

[0116] Treating ρn as a whole N, we have I = LN, where n is a unit vector and L is an M×3 matrix. When M > 3, the matrix is ​​not full rank. We use the least squares method to obtain an approximate solution, thus obtaining the surface normal vector map of the object. In this embodiment, four light sources are used, and the L matrix is ​​a 4×3 matrix with non-full rank. We use the least squares method to obtain an approximate solution, thereby obtaining the surface normal vector map of the object.

[0117] L T I = L T LN

[0118] N=(L T L) -1 L T I

[0119]

[0120] As a further optional implementation, when the surface equation is transformed into the explicit function f(x,y)-z=0, the normal vector N=(N x N y N z ) can be represented as N = (f x ,f y ,-1), where Obtain the gradients along the x-direction and y-direction of the object's surface, and then calculate the corresponding divergence and curvature plots using the divergence and curvature formulas:

[0121]

[0122]

[0123] In the formula, p is f x q is f yDiv and K g It represents the divergence and curvature at various locations of an object, used to describe the object's geometric shape information.

[0124] S5. Input the fused image and scatter plot into the preset dual-branch feature fusion network to extract the fused features of the image and output the final defect detection result.

[0125] In the dual-branch feature fusion network, the backbone responsible for feature extraction contains two branches. One branch takes as input an enhanced image fused from four RGB images, and the other branch takes as input an object surface scatter map. A CSP structure is used as the information extraction module of the backbone network. A residual spatial channel attention module is introduced after each CSP structure, enabling the network to adaptively learn the weight values ​​of different regions and channels of the feature map, thereby better highlighting effective information and suppressing background interference. The expression for the attention module is:

[0126] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0127] M s (F)=σ(Conv(Cat[AvgPool(F),MaxPool(F)]))

[0128] F out =F + F·M c (F)+F·M s (F)

[0129] In the formula, M c For channel attention maps, M s Here, F represents the spatial attention map, MLP represents the shared multilayer perceptron, AgvPool and MaxPool represent average pooling and max pooling, respectively, Cat represents the stacking operation, and Conv represents the convolution operation.

[0130] As an optional implementation, this dual-branch feature fusion network is an improved dual-branch YOLOv5s feature fusion network. In the original YOLOv5s network, it is necessary to fuse the shallow high-resolution weak semantic feature map with the deep low-resolution strong semantic feature map. Here, the improved dual-branch network first stacks the feature information extracted by the two branches on the channel, and then uses a CBS structure with half the number of channels to achieve information fusion of the two branches. Then, in the neck part, the PAnet structure is used to further fuse with the deep feature map information, and in the head part of the detection head, all features are integrated to complete the final detection task. The multi-task loss includes classification loss, target confidence loss, and bounding box regression loss.

[0131] Specifically, the four acquired images can present the defect morphology under different lighting conditions. The fused image further enhances the defect features. The fused image is used as the input of one of the branches, which mainly provides the network with light reflection information of the object surface. This branch is easily affected by lighting conditions. Therefore, a branch is added to the backbone of the network to input the object surface scatter map, which is mainly responsible for providing the geometric shape information of the object surface. This branch is more robust. The two branches can extract their respective effective information, reducing crosstalk.

[0132] The backbone introduces a residual spatial channel attention module after each CSP structure. Referring to the CBAM structure, the spatial attention performs average pooling and max pooling in the channel dimension and then concatenates them. The concatenated feature map is then convolved to obtain the spatial attention map. The channel attention performs global average pooling and global max pooling based on width and height. The features are fed into a multilayer perceptron, and the output features are summed to obtain the final channel attention map. The original features are multiplied by the spatial attention map and the channel attention map respectively to obtain the weighted features, which are then summed with the original features to obtain the residual spatial channel attention structure.

[0133] The original YOLOv5S network consists of a CSPDarknet backbone, a PAN neck, and a multi-scale prediction head. The backbone network is mainly responsible for extracting and integrating shallow features to form deeper and higher-level semantic features. The PAN neck structure fuses shallow location information with deep semantic features to enhance the ability to identify and locate targets at multiple scales. The improved network in this application uses two branches in the backbone. Therefore, when fusing with deep semantic features, the features extracted by the two branches need to be fused beforehand. Here, the features of the two branches are stacked on the channels, and then the CBS structure with half the number of channels is used to achieve information fusion of the two branches. Then, it is further fused with the subsequent deep semantic features, and finally input into the multi-scale detection head to achieve the final defect identification and localization.

[0134] The following detailed explanation, in conjunction with the accompanying drawings and specific examples, will be provided.

[0135] This embodiment uses cold-rolled phosphated steel sheet as the inspection object and introduces a surface defect detection method based on photometric stereo and deep learning, including the following steps:

[0136] Step 1: As Figure 3 As shown, Figure 3The aforementioned photometric stereo image acquisition device comprises four strip light sources mounted on the crossbeams of an aluminum profile frame in four directions. A stage is positioned in the central area, an industrial camera is located directly above the stage, and a calibration sphere is placed on the stage. The four strip light sources are illuminated sequentially from different directions to obtain four corresponding images. The center point of the highlight area of ​​the calibration sphere in each image is identified, based on the relationship between incident light, reflected light, and the normal vector. Figure 2 The diagram illustrates the vector relationship, from which the illumination direction matrix is ​​obtained.

[0137] Step 2: Keeping the light source and camera positions unchanged, acquire 156 sets of images of the steel plate, of which 124 sets are used as the training set and 32 sets are used as the test set. Each set contains images from four lighting directions, with an image resolution of 1200×1200. The images are then uniformly resized to 640×640 to fit the network model input. The contrast pyramid algorithm described above is used to fuse and enhance the four images, as shown below. Figure 6 As shown, it serves as the input for one of the branches of the model.

[0138] Step 3: Using the classic photometric stereo algorithm, calculate the surface normal map of the object based on the four images and the illumination direction matrix, and then obtain the gradient, divergence, and curvature at various locations on the object's surface, such as... Figure 7 As shown, Figure 7 This section presents the scatter plot and Gaussian curvature plot of the object's surface. Here, the scatter plot is chosen to describe the geometric shape information of the object's surface.

[0139] Step 4: Figure 5 The diagram shows an improved dual-branch YOLOv5s feature fusion network structure. The fused image is used as input to one branch, and the divergence image as input to the other branch. Scaling, flipping, and mosaic enhancement are applied before input to the network to augment the dataset. The batch size is set to 8, and the SGD optimizer is used. The loss function consists of three parts: bounding box regression loss, classification loss, and target confidence loss. The bounding box loss uses CIOU loss, while the classification and target confidence losses use BCE loss. Iterative training is performed for 400 epochs. The improved network structure is shown below. Figure 5 As shown, the backbone uses two branches. The Attention module in each branch is a residual spatial channel attention. The spatial attention performs average pooling and max pooling in the channel dimension to obtain the feature map. and The data is then stacked along the channels, and a standard convolution with a kernel of 7×7 and padding of 3 is used, along with a sigmoid function, to obtain the spatial attention map M. s Channel attention is obtained by performing global average pooling and global max pooling in the spatial domain. and The data is then fed into a multilayer perceptron, with the number of hidden nodes in the multilayer perceptron being C / r, and the reduction ratio r being 16. Finally, the outputs are combined and passed through a sigmoid function to obtain the channel attention map M. c The original features are multiplied by the spatial attention map and the channel attention map respectively to obtain the weighted features. These weighted features are then summed with the original features to obtain the residual spatial-channel attention structure, expressed as:

[0140] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0141] M s (F)=σ(Conv(Cat[AvgPool(F),MaxPool(F)]))

[0142] F out =F + F·M c (F)+F·M s (F)

[0143] The features extracted from the two branches are fused using the Fusion module. The Fusion module stacks the features from the two branches along the channels and then merges the feature information using a CBS structure with half the number of channels. The formula is as follows:

[0144] F fus =Conv(Cat(F) div ,F Ori ))

[0145] The neck PAN structure fuses shallow location information with deep semantic features, forming three prediction branches in the final prediction head, which are used to detect large, medium and small targets respectively. The prediction head is a 1×1 convolutional kernel, which reduces the number of channels to 3×(nc+4+1), where nc is the number of categories, 4 represents the width, height and position coordinates of the bounding box, and 1 represents the target confidence.

[0146] To verify the effectiveness and gain of the improved network, a comparative experiment was conducted. The experimental environment was based on a Windows 10 system, a GeForce 3060ti graphics card with 8GB of video memory, an Intel i5-11400 processor, and the PyTorch 1.11.0 deep learning library. The configuration for the comparative experiment is as follows.

[0147] (1) Single branch

[0148] a. Single RGB image as input

[0149] b. Fuse the image and scatter plot stack as input

[0150] (2) Two branch roads

[0151] a. The fused image and scatter plot are input via two separate branches.

[0152] Figure 6 The diagram shows some case studies from the comparative experiments. The first column shows the detection results for a single RGB image with single-branch input; the second column shows the detection results for a fused image and a scatter plot with stacked input; and the third column shows the detection results for a fused image and a scatter plot with dual-branch input. The experimental results show that using a single RGB image as input may misdetect pit defects as protruding defects. This is because the network can only determine the defect type by observing the shadows formed by the depth changes in the defect area in the RGB image. Since the shadow changes around pits and protrusions are highly similar, misdetection is likely. Furthermore, due to the low contrast, some pit and protruding defects may be missed. However, in the scatter plot, pits and protruding defects have high contrast and are clearly distinguishable. Figure 8 As shown, when the fused image and scatter map are stacked as input, false detections do not occur, but missed detections still occur. This is because simply stacking images may cause crosstalk between images of different modalities, weakening the effective information between them and affecting the model's extraction of defect features. The dual-branch feature fusion network used in this embodiment takes the scatter map and the fused image as input, and the two branches extract features from the scatter map and the fused image respectively. The detection results obtained do not show false detections or missed detections, and the detection effect for pit protrusions is better than the former two.

[0153] In summary, compared with the prior art, the method of this embodiment has at least the following advantages and benefits:

[0154] (1) The method provided by this invention uses multi-directional lighting to highlight the defect area, avoiding the problem that traditional lighting only uses a single direction, causing the defect to be hidden in the background. It is especially suitable for defects with obvious directionality, such as scratches. In addition, the surface divergence map of the object calculated by photometric stereo method and divergence formula can characterize the defect information from the perspective of geometric morphology, effectively enhancing the low contrast of defects such as pits and protrusions, thereby reducing the difficulty of subsequent model detection.

[0155] (2) The present invention performs contrast pyramid fusion on images under different lighting conditions, thereby extracting and fusing effective information from multiple images into one image, enhancing the contrast of scratches and abrasions, and making the overall brightness of the image more uniform. This allows the model to fully acquire information from four images during subsequent input, while reducing computation and improving the model's inference speed.

[0156] (3) This invention improves the general YOLOv5s network into a dual-branch feature fusion network, extracting image features from the original RGB image and the scatter plot input in two parallel branches, which solves the information crosstalk problem caused by directly stacking images as input in existing methods. At the same time, the residual space channel attention module is introduced, which enables the network to adaptively increase the weight of the defect region and shield the background information, thereby improving the defect detection accuracy and reducing false detections and false negatives.

[0157] This embodiment also provides a metal surface defect detection device, including:

[0158] At least one processor;

[0159] At least one memory for storing at least one program;

[0160] When the at least one program is executed by the at least one processor, the at least one processor performs the following: Figure 1 The method shown.

[0161] This embodiment of the metal surface defect detection device can execute the metal surface defect detection method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0162] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0163] This embodiment also provides a storage medium storing instructions or programs that can execute the metal surface defect detection method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0164] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0165] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0166] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0168] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0169] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0170] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0171] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0172] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for detecting defects on metal surfaces, characterized in that, Includes the following steps: A photometric stereo image acquisition device is used to acquire images of a high-gloss metal calibration sphere under different illumination directions, and the illumination direction matrix is ​​obtained based on the acquired images and calibration formula; wherein, the photometric stereo image acquisition device includes multiple light sources in different directions; The metal to be tested is placed in the photometric stereo image acquisition device, and each light source is lit in sequence to obtain a sequence of object surface images under different lighting directions. Multiple object surface images in the object surface image sequence are fused to obtain a fused image; Calculate the scatter map of the object surface based on the object surface image sequence and the illumination direction matrix; The fused image and scatter plot are respectively input into a preset dual-branch feature fusion network to extract the fused features of the image and output the final defect detection result. The step of calculating the scatter map of the object surface based on the object surface image sequence and the illumination direction matrix includes: Based on the object surface image sequence and illumination direction matrix, the divergence map and curvature map of the object surface are calculated using the photometric stereo method, divergence formula, and curvature formula. The calculation formula for the photometric stereo method is as follows: In the formula, I This represents the brightness value at a specific location in the image. L For the illumination direction matrix, The surface reflectance of the object. n This is the unit normal vector at that location; Will View as a whole N Then the formula for calculating the normal vector is: 。 2. The method for detecting metal surface defects according to claim 1, characterized in that, The photometric stereo image acquisition device includes: An industrial camera; Four light sources; A light source frame is used to mount the light source, and the four light sources are respectively mounted on the crossbeams in four directions of the light source frame; the four light sources form a non-connected aperture. A stage, located in the center of the aperture, is used to place the metal to be measured; A camera bracket is used to fix the industrial camera, which is located directly above the stage and shoots vertically downwards.

3. The method for detecting metal surface defects according to claim 1, characterized in that, The expression for the calibration formula is: In the formula, N Let be the normal vector at the center point of the highlight region. V The reflected light vector, For each individual light source, the illumination direction vector is represented by a single light source. The illumination direction vectors of all light sources are then combined into an illumination direction matrix.

4. The method for detecting metal surface defects according to claim 1, characterized in that, The step of fusing multiple object surface images from the object surface image sequence includes: The contrast pyramid fusion algorithm is used to merge multiple images into one image; The contrast pyramid fusion algorithm first obtains the Laplacian pyramid of the image, calculated using the following formula: In the formula, It is a Gaussian filter. For the first Gaussian pyramid l Layer image, For the first l The image after interpolation and augmentation of the layer image, where Z is the highest layer number of the Laplace pyramid. For the first of Laplace's pyramids l Layered images; Will See as Against a background of [unclear], the contrast pyramid is defined as follows: In the formula, For the k-th image l The value at position x in the layer contrast pyramid. , The values ​​at corresponding positions of the contrast pyramid and Gaussian pyramid after fusion are represented respectively. A new contrast pyramid is constructed by taking the value with the largest absolute value at that position from M images, and a new Gaussian pyramid is constructed by taking the average value of the Gaussian pyramids from M images. The fused image is then restored using the new contrast pyramid and the Gaussian pyramid.

5. The method for detecting metal surface defects according to claim 1, characterized in that, The expression for the divergence formula is: The expression for the curvature formula is: In the formula, p for , indicating the surface of an object along x Gradient of direction, q for , indicating the surface along y Gradient of direction, Div It represents the divergence at various locations of an object. It represents the curvature at various locations of an object.

6. The method for detecting metal surface defects according to claim 1, characterized in that, The dual-branch feature fusion network is a dual-branch YOLOv5s feature fusion network, which introduces a residual space channel attention module, as shown in the following expression: In the formula, , These represent the channel attention map and the spatial attention map, respectively. F As input features, It is the Sigmoid activation function. MLP To share multilayer perceptrons, For stacking operations, AvgPool and MaxPool represent average pooling and max pooling, respectively. For convolution operations, For output features.

7. The method for detecting metal surface defects according to claim 1, characterized in that, The dual-branch feature fusion network performs feature fusion on the features extracted from the two branches using the following formula: In the formula, Features extracted from the branches of the scatter plot. Features extracted from RGB image branches Conv A convolution operation with half the number of channels. It is a dual-branch fusion feature. This is a stacking operation.

8. A metal surface defect detection device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-7.

Images