Metal diaphragm surface defect detection method based on cross feature fusion

Abstract

The present invention discloses a metal diaphragm surface defect detection method based on cross-feature fusion. The method comprises: establishing a metal diaphragm surface defect dataset; training a metal diaphragm surface defect detection model; simultaneously inputting a group of images from the training set into a defect detection network; extracting features through a visual converter backbone network; fusing the features using an improved cross-feature pyramid module; obtaining a detection candidate frame from the fused features through a region generation network module; locating and classifying defect targets within the candidate frame using a region of interest detection head module; iteratively optimizing model parameters to form a detection model for metal diaphragm surface defects; and continuously photographing metal diaphragms in actual production using multiple cameras. The captured related image groups are input into the defect detection model to detect defects in the images. The present invention uses a visual converter and an improved cross-feature pyramid module to efficiently and accurately implement metal diaphragm surface defect detection tasks.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and in particular relates to a metal diaphragm surface defect detection method based on cross-feature fusion. Background Art

[0002] The metal diaphragm in an aircraft propulsion system tank is a critical component that suppresses propellant sloshing during attitude adjustments and maneuvering maneuvers. Its surface quality directly impacts the diaphragm's fatigue resistance and its operational performance, including deformation and rollover. Therefore, the diaphragm's material, configuration, geometric parameters, and manufacturing process are held to extremely high standards. For example, the diaphragm surface requires no visible defects such as cracks, delamination, fissures, rust, spots, or residual raw material. Scratches or indentations deeper than 4μm are also prohibited. This not only places extremely high demands on the manufacturing process but also presents significant challenges for product quality inspection. Furthermore, the metal diaphragm surface has a high reflectivity, making it difficult for standard lighting and image acquisition systems to capture high-contrast images of defects. Therefore, defect detection on the metal diaphragm surface is crucial for improving production quality.

[0003] Traditional machine vision-based defect detection algorithms include matching difference, threshold segmentation, support vector machines, and more. While these traditional image processing methods can effectively detect defect images in specific environments, they are sensitive to noise when image capture conditions vary or when the background is complex, making it difficult to achieve high-precision defect detection. Summary of the Invention

[0004] To solve the above problems, the present invention discloses a metal diaphragm surface defect detection method based on cross-feature fusion, applies a visual converter network to the defect detection algorithm model to improve the model's defect feature extraction capability; based on the defect characteristics of actual metal diaphragm surface defects, the feature fusion network is improved specifically, and a cross-feature fusion network is proposed to enhance the detection model's ability to fuse defect features; combined with a multi-illumination image channel fusion detection strategy, metal diaphragm surface defects are detected efficiently, accurately and reliably.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] A metal diaphragm surface defect detection method based on cross-feature fusion includes the following steps:

[0007] S1: Use multiple cameras and different light sources to collect images of the metal diaphragm surface. Multiple images collected in the same area of ​​the metal diaphragm are grouped together to establish a corresponding relationship. The locations and types of metal diaphragm surface defects in the collected images are manually annotated to form a metal diaphragm surface defect dataset, which is divided into a training set and a validation set.

[0008] S2, inputting the training samples in the training set into the metal diaphragm surface defect detection feature extraction network; said step S2 further comprises:

[0009] S21: The image in the training set and the corresponding multiple images are simultaneously used as input according to different channels. The annotation information of the input images in different channels is processed by non-maximum suppression in the channel space to remove highly repeated annotation information. The highly overlapping label information between different image channels is removed, and the processed image group is input into the visual converter backbone network;

[0010] S22, based on the image group processed in step S21, the image group will be input into the visual converter backbone network according to the channel to further extract features to obtain the feature map of the defect image. The visual converter backbone network includes multiple different stages, and different feature maps will be generated at different stages. The length and width of the feature maps are halved step by step, forming a hierarchical structure that decreases step by step in space. The feature map information of different stages can be further input into the feature pyramid module. The visual converter backbone network divides the image into multiple window areas for serialization operation. Adjacent windows overlap with each other to preserve the local continuity of the image. The serialization operation converts the original two-dimensional image into a one-dimensional vector. When the input image is high H, wide W, and has C channels, the serialization operation serializes the input image into N image blocks. The image block size is:

[0011] P×P×C

[0012] Where P is the height and width of the image block. The two-dimensional image is expanded into a one-dimensional vector through serialization operation, where the norm of the one-dimensional vector is:

[0013] N×P 2 C

[0014] The one-dimensional vector is input into the converter encoder module. The visual converter backbone network controls the output size of the backbone network feature map through a gradual change strategy of the image block parameter P, where the image block size of the i-th stage is defined as P i , for the input feature map F of the i-th stage i-1 , can be decomposed into multiple image blocks, the number of which is:

[0015] (H i-1 ×W i-1 ) / P i 2

[0016] Each image block is then converted into a one-dimensional vector and projected onto C i Dimension, that is, the output feature map size dimension of the i-th stage is:

[0017] (H i-1 / P i )×(W i-1 / P i )×C i

[0018] By flexibly changing the image block size P, the output of feature maps of different scales can be achieved;

[0019] Compared to ordinary backbone networks, the visual converter backbone network has better cross-scale feature representation capabilities by controlling the image block size to obtain feature maps of different scales. Using feature maps at different levels to obtain feature information at different scales enables the network to better process details and global information in the image. Compared to ordinary backbone networks that require a large number of convolution and pooling layers to extract features, the visual converter backbone network uses converters and utilizes its self-attention mechanism to learn the dependencies between features. By calculating the similarity between different image blocks, it further learns the dependencies between different image blocks. This also ensures that the visual converter backbone network always has a global receptive field, unlike the local receptive field of ordinary backbone networks that increases layer by layer. It can be understood that the visual converter backbone network can better extract defect features in the image than ordinary backbone networks and further output them to the subsequent feature pyramid module.

[0020] S3, based on the multi-scale feature information processed in step S2, further inputs the cross feature pyramid module:

[0021] The feature maps of different scales processed in step S2 are further input into the improved cross feature pyramid module for feature fusion. The improved cross feature pyramid module includes a bottom-up path, a top-down path, a lateral connection path, and a cross fusion path. The improved cross feature pyramid module performs feature dimensionality reduction and dimensionality increase on feature maps of different resolutions, and merges the feature maps of corresponding resolutions to generate a merged feature map. The original input image and the feature maps of different scales generated after step S2 are regarded as the bottom-up path in the improved cross feature pyramid module. Some feature maps in the bottom-up path are selected as the overall input of the improved cross feature pyramid module. The feature maps are respectively subjected to preliminary feature fusion through the lateral connection path and the cross fusion path to obtain a first-level intermediate feature map. The lateral connection path uses a 1×1 convolutional layer for connection, and the output dimension is converted during the lateral connection. The cross fusion path uses an improved upsampling module, which uses the nearest neighbor interpolation method and fuses 1×1 convolutional layers, so that the upsampling module can better achieve a balance between the dimensions of feature maps of different scales.

[0022] After completing the initial feature fusion to obtain the first-level intermediate feature map, the feature map of the defect image is further fused through the horizontal connection path and the cross fusion path to obtain the second-level intermediate feature map. The cross fusion path uses an F×F convolution layer with a step size parameter of S and a padding parameter of P to achieve overall downsampling of the first-level intermediate feature map. The calculation formula for the feature map size after overall downsampling is as follows:

[0023] N=(W-F+2×P) / S+1

[0024] Where N represents the output feature map size, W represents the input feature map size, F represents the convolution kernel size, P represents the padding parameter, and S represents the step size parameter.

[0025] After the downsampling operation, the eigenvalues ​​in the feature map can be calculated using the following formula:

[0026]

[0027] x=i+s h (m-1)-p h

[0028] y=j+s w (n-1)-p w

[0029] Among them, O(i,j) represents the eigenvalue of position (i,j) in the output feature map, I(x,y) represents the eigenvalue of position (x,y) in the input feature map, and K(m,n) represents the weight of position (m,n) in the convolution kernel. hand s w Represents the step size of the convolution kernel in the row and column directions, p h and p w Represents the number of zero padding added in the height and width directions of the input feature map respectively.

[0030] On the basis of the secondary intermediate feature maps, horizontal connections are performed, and upsampling and feature fusion operations are performed step by step to obtain the output feature maps. Finally, the output feature maps pass through the output convolution layer to achieve the final output;

[0031] Compared with the ordinary feature pyramid module, the improved cross feature pyramid module adds a cross fusion path between adjacent feature maps, so that the feature map information is no longer limited to the current feature layer. The cross fusion between adjacent feature layers allows the model to extract more semantic information of adjacent feature layers, so that the model can pay more attention to the relationship between adjacent feature layers. The fusion between adjacent feature layers makes the model more likely to find useful defect feature information in the fused feature map. This is also because most defects have local invariance, that is, when a part of a larger defect is intercepted, this part of the defect can still be regarded as an independent defect body rather than just a part of the original defect. Therefore, the improved feature pyramid module pays more attention to the feature fusion between adjacent feature layers, which can better achieve accurate defect judgment. After the feature fusion of the cross fusion path, the multi-scale feature map information is further combined through the top-down connection path to finally achieve a high degree of fusion of defect features.

[0032] S4, based on the fusion feature information processed in step S3, further inputs the region generation network module and the region of interest detection head module, and the step S4 further includes:

[0033] S41: Based on the fusion features obtained in step S3, the detection candidate boxes are obtained through the region generation network module. The region generation network module detects the areas in the feature map where the detection object may exist and forms candidate boxes. In actual operation, a sliding convolution window is used to slide the entire feature map. Anchor boxes of different sizes and proportions are used for each window position. The probability of the existence of defects in the anchor boxes is calculated, and the bounding box regression parameters are calculated. A large number of defect target candidate boxes are generated. After non-maximum suppression processing and elimination of a large number of overlapping candidate boxes, these candidate boxes are input into the region of interest detection head module.

[0034] S42: Based on the candidate box obtained in step S3, the candidate box is input into the region of interest detection head module to further locate and classify the defect targets in the candidate box. The model detection results are compared with the manual labeling results, and the model loss function is calculated. The detection model parameters are continuously optimized iteratively based on the loss value, and finally a target detection model for metal diaphragm surface defects is formed. The region generation network module and the region of interest detection head module share the fusion features obtained in step S3.

[0035] S5, continuously photographing a metal diaphragm workpiece in actual production, inputting the images of the same area taken by multiple cameras into the metal diaphragm surface defect detection model trained and generated by S2 to S4 according to the channel, detecting defects in the multi-channel images, and marking the location and type of metal diaphragm surface defects appearing in the images with rectangular frames;

[0036] Beneficial effects of the present invention:

[0037] 1. The metal diaphragm surface defect detection method has been improved. By using multiple sets of images under different lighting conditions, the problem of metal diaphragm surface defect detection being difficult to highlight in images under a single lighting condition has been solved.

[0038] 2. The defect detection network's feature extraction capabilities are enhanced through a visual transformer backbone network. This addresses the challenges of efficient and accurate detection of metal diaphragm surface defects due to insufficient feature extraction caused by the diversity of defect types and scales. By simultaneously feeding multiple defect images into the defect detection network on a channel-by-channel basis, feature fusion is achieved at the image input level. This expands the traditional single-image defect detection strategy to a fused detection strategy for multiple images, improving actual defect detection accuracy.

[0039] 3. Improve the feature pyramid module and add a cross-fusion path between adjacent feature maps. This makes the model's feature map information for defects no longer limited to the current feature layer. Through the cross-fusion of feature information between adjacent feature layers, the model can pay more attention to the semantic information of itself and adjacent feature layers to achieve more accurate defect judgment. After the feature fusion of the cross-fusion path, the final output of the fused feature is achieved through a top-down connection path, so that the detection model can better fuse the multi-scale features extracted by the visual converter backbone network, improve the model feature fusion rate, and further improve the actual defect detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Flowchart of a metal diaphragm surface defect detection method based on cross-feature fusion according to a specific embodiment of the present application;

[0041] Figure 2Schematic diagram of the structure of the overall defect detection network according to an embodiment of the present application;

[0042] Figure 3 Schematic diagram of the structure of the converter encoder module in the visual converter backbone network according to an embodiment of the present application;

[0043] Figure 4 Schematic diagram of the structure of an improved feature pyramid module according to an embodiment of the present application;

[0044] Figure 5 Schematic diagram of the system structure for acquiring images of the metal diaphragm surface using a multi-light source system;

[0045] Figure 6 This is the surface image of the metal diaphragm collected using a multi-light source system.

[0046] Figure 7 Schematic diagram of common defects on the surface of metal diaphragms obtained using a multi-light source system. DETAILED DESCRIPTION

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0048] In order to address the problems of defects in images being difficult to highlight during metal diaphragm surface inspection, as well as insufficient defect feature extraction caused by the diversity of defect types and defect scales, which leads to the inability to achieve efficient and accurate detection of metal diaphragm surface defects, the present application provides a metal diaphragm surface defect detection method based on cross-feature fusion. In this method, a metal diaphragm defect detection model is obtained through training through a visual converter backbone network, a feature pyramid module, a region generation network module, and a region of interest detection head module to accurately detect metal diaphragm surface defects. The visual converter backbone network is used to improve the network's ability to extract defect features. By simultaneously inputting multiple images into the detection network by channel, the detection network's ability to detect metal diaphragm surface defects under multiple lighting conditions is improved, thereby avoiding problems such as insufficient defect detection accuracy caused by the lack of feature information in a single image, and improving the actual defect detection accuracy.

[0049] Specifically, Figure 1 This is a flow chart of a metal diaphragm surface defect detection method based on cross-feature fusion according to a specific embodiment of the present application. The specific steps are as follows:

[0050] S1, use multiple cameras and different light sources to collect images of the metal diaphragm surface respectively, record multiple images collected in the same area of ​​the metal diaphragm as a group, establish a corresponding relationship, manually annotate the location and type of metal diaphragm surface defects in the collected images, form a metal diaphragm surface defect dataset, and divide it into training set and validation set.

[0051] S2, input the training samples in the training set into the metal diaphragm surface defect detection model, Figure 2 The overall process of the defect detection model is shown; the step S2 further includes:

[0052] S21, the pictures in the training set and the corresponding multiple pictures are used as inputs at the same time according to different channels, and the annotation information of the input pictures in different channels is subjected to non-maximum suppression in the channel space to eliminate highly repeated annotation information. Specifically, the manual labels of the three pictures in the same group are input into the first, second, and third image channels of the defect detection network respectively, that is, non-maximum suppression is performed on the labels between different images at the channel level, and the highly overlapping label information between different images is eliminated, while the image label information in the same image channel is not compared. When performing the non-maximum suppression operation, it is necessary to calculate the intersection-over-union (IOU) of the two defect annotation boxes of the same type. The calculation formula is:

[0053] IOU=Area(A∩B) / Area(A∪B)

[0054] Where A and B represent two different marked areas respectively. Let the coordinates of the upper left corner and the lower right corner of the rectangular area marked by A be (X 11 ,Y 11 )、(X 12 ,Y 12 ), the coordinates of the upper left corner and the lower right corner of the rectangular area marked by B are (X 21 ,Y 21 )、(X 22 ,Y 22 );

[0055] The overlapping area is:

[0056] Area(A∩B)=abs(X 12 -X 21 )×abs(Y 12 -Y 21 )

[0057] Union area:

[0058] Area(A∪B)=abs(X 11 -X 12 )×abs(Y 11 -Y 12)+abs(X 21 -X 22 )×abs(Y 21 -Y 22 )-abs(X 12 -X 21 )×abs(Y 12 -Y 21 ) where abs represents the absolute value operation. In practice, the image annotation information input from the first image channel is first used as the reference to mark the rectangular box. The image annotation information input from the second and third image channels is then used to mark the rectangular box. If the IOU of the marked rectangular box with the reference marked rectangular box exceeds a certain threshold, it is removed. Then, the image annotation information input from the second image channel is used as the reference to mark the rectangular box. The image annotation information input from the third image channel is then used to mark the rectangular box. This process is repeated. The processed image group is input into the visual transformer backbone network.

[0059] S22: The image group processed based on step S21 will be simultaneously input into the visual converter backbone network by channel to further extract features to obtain the feature map of the defect image. It includes 4 different stages, and different feature maps will be generated at different stages. The feature maps obtained in the four stages are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size, respectively, forming a hierarchical structure that decreases step by step in space. This allows the feature map information of different stages to be further input into the feature pyramid module for further feature fusion. The visual converter backbone network divides the image into multiple window areas for serialization operations. Adjacent windows overlap with each other to preserve the local continuity of the image. The serialization operation converts the original two-dimensional image into a one-dimensional vector. Assume that the input image dimension is:

[0060] H×W×C

[0061] Where H, W, and C represent the height, width, and number of channels of the image, respectively.

[0062] The serialization operation transforms the input image into N image blocks with the size of:

[0063] P×P×C

[0064] Where P is the height and width of the image block.

[0065] Expand the two-dimensional image into a one-dimensional vector, where the norm of the one-dimensional vector is:

[0066] N×P 2 C

[0067] In order to retain the local position information of the image, there should be a certain amount of information redundancy between different image blocks during sampling, that is, there should be a certain amount of overlap between adjacent sampled image blocks. The obtained one-dimensional vector is input into the converter encoder module. The visual converter backbone network controls the output size of the backbone network feature map through the gradual change strategy of the image block parameter P, where the image block size of the i-th stage is defined as P i , for the input feature map F of the i-th stage i-1 , can be decomposed into multiple image blocks, the number of which is:

[0068] (H i-1 ×W i-1 ) / P i 2

[0069] Each image block is then converted into a one-dimensional vector and projected onto C i Dimension, that is, the output feature map size dimension of the i-th stage is:

[0070] (H i-1 / P i )×(W i-1 / P i )×C i

[0071] By flexibly changing the image block size P, the output of feature maps of different scales can be achieved;

[0072] It can be understood that when an H×W image is input into the backbone network, it is first divided into multiple image blocks, the number of which is:

[0073] H×W / (4×4)

[0074] The size of each image block is 4×4. After linear projection of all images and adding position encoding, the feature map is generated with a size of:

[0075] H / 4×W / 4

[0076] The channel is the feature map of C1, repeat the above operation to generate the size of

[0077] H / 8×W / 8

[0078] H / 16×W / 16

[0079] H / 32×W / 32

[0080] It is understandable that the embodiment of the present application uses a visual converter backbone network to extract image features, wherein the visual converter backbone network is composed of a converter encoder module, which can better extract the surface defect features of the metal diaphragm. Specifically, the converter encoder module structure is as follows Figure 3 As shown. The serialized sequence vector is input into the converter encoder module, and after regularization, it enters the multi-head attention module. The multi-head attention module obtains multiple sets of different linear projections through independent learning to transform the query value, key value, and value. These multiple sets of transformed query values, key values, and values ​​will be attention pooled in parallel. Finally, the outputs of these multiple attention pools are spliced ​​together and transformed through another learnable linear projection to produce the final output. The output result is added to the original input and then regularized and input into the multi-layer perceptron. The multi-layer perceptron introduces multiple hidden layers between the input layer and the output layer. It is a fully connected network, which is also called a deep feedforward network. Its output is added to its unregularized input and input into the next layer. The above steps are repeated many times to form the converter encoder module in the visual converter backbone network.

[0081] Compared to ordinary backbone networks, the visual converter backbone network has better cross-scale feature representation capabilities by controlling the image block size to obtain feature maps of different scales. Using feature maps at different levels to obtain feature information at different scales enables the network to better process details and global information in the image. Compared to ordinary backbone networks that require a large number of convolution and pooling layers to extract features, the visual converter backbone network uses converters and utilizes its self-attention mechanism to learn the dependencies between features. By calculating the similarity between different image blocks, it further learns the dependencies between different image blocks. This also ensures that the visual converter backbone network always has a global receptive field, unlike the local receptive field of ordinary backbone networks that increases layer by layer. It can be understood that the visual converter backbone network can better extract defect features in the image than ordinary backbone networks and further output them to the subsequent feature pyramid module.

[0082] S3: Based on the feature maps of different scales processed in step S2, the feature maps are further input into an improved cross feature pyramid module to perform feature fusion on the feature maps. The improved cross feature pyramid module includes a bottom-up path, a top-down path, a lateral connection path, and a cross fusion path.

[0083] The overall module architecture is as follows Figure 4 As shown in the figure, I1 represents the original input image, I2~I5 represent the feature maps of different scales generated after the four different stages of the visual converter backbone network, and the sizes of the feature maps are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image respectively. They are regarded as the bottom-up pathways in the improved cross feature pyramid module, and I3~I5 are selected as the overall input of the improved feature pyramid module, as shown in Figure 4 As shown, I3 to I5 are respectively subjected to preliminary feature fusion through the horizontal connection path and the cross fusion path to obtain the first-level intermediate feature map M 3-1 ~M 5-1, where the horizontal connection path uses a 1×1 convolutional layer to connect, and the output dimension is transformed while the horizontal connection is completed, I3~I5 to M 3-1 ~M 5-1 The cross-fusion path uses an improved upsampling module. On the basis of upsampling using nearest neighbor interpolation, a 1×1 convolution layer is added, so that the upsampling module can better achieve the balance between the dimensions of feature maps of different scales. The coordinate transformation formula of the nearest neighbor interpolation is as follows:

[0084] srcX=dstX×(srcW / dstW)

[0085] srcY=dstY×(srcH / dstH)

[0086] In the formula, dsrX and dsrY are the horizontal and vertical coordinates of a pixel in the target image, dstW and dstH are the height and width of the target image, and srcW and srcH are the width and height of the original image. (srcX, srcY) is the original image coordinate corresponding to the target image point (dstX, dstY).

[0087] After completing the initial feature fusion, we get the first-level intermediate feature map M 3-1 ~M 5-1 After that, the feature map of the defect image is further fused through the horizontal connection path and the cross fusion path, such as Figure 4 Shown M 3-1 ~M 5-1 The second-level intermediate feature map M is obtained through the lateral connection path and the cross fusion path. 3-2 ~M 5-2 , where the cross-fusion path uses a 3×3 convolutional layer with a stride parameter of 2 and a padding parameter of 1 to achieve overall downsampling of the first-level intermediate feature map. The calculation formula for the feature map size after overall downsampling is as follows:

[0088] N=(W-F+2×P) / S+1

[0089] Where N represents the output feature map size, W represents the input feature map size, F represents the convolution kernel size, P represents the padding parameter, and S represents the step size parameter.

[0090] After the downsampling operation, the eigenvalues ​​in the feature map can be calculated using the following formula:

[0091]

[0092] x=i+s h (m-1)-p h

[0093] y=j+s w (n-1)-pw

[0094] Among them, O(i,j) represents the eigenvalue of position (i,j) in the output feature map, I(x,y) represents the eigenvalue of position (x,y) in the input feature map, and K(m,n) represents the weight of position (m,n) in the convolution kernel. h and s w Represents the step size of the convolution kernel in the row and column directions, p h and p w Represents the number of zero padding added in the height and width directions of the input feature map respectively.

[0095] In the secondary intermediate feature map M 3-2 ~M 5-2 On the basis of , they are horizontally connected, and up-sampling and feature fusion operations are performed step by step to obtain output feature maps P3~P5. Finally, the output feature maps P3~P5 pass through the output convolution layer with a convolution kernel size of 3×3, a step parameter of 1, and a padding parameter of 1 to achieve the final output Output1~Output3.

[0096] Compared with the ordinary feature pyramid module, the improved cross feature pyramid module adds a cross fusion path between adjacent feature maps, so that the feature map information is no longer limited to the current feature layer. Through the cross fusion between adjacent feature layers, the model can extract more semantic information of adjacent feature layers, so that the model can pay more attention to the relationship between adjacent feature layers. The fusion between adjacent feature layers makes the model more likely to find useful defect feature information in the fused feature map. This is also because most defects have local invariance, that is, when a part of a larger defect is intercepted, this part of the defect can still be regarded as an independent defect body rather than just a part of the original defect. Therefore, the improved feature pyramid module pays more attention to the feature fusion between adjacent feature layers, which can better achieve accurate defect judgment. After the feature fusion of the cross fusion path, the multi-scale feature map information is further combined through the top-down connection path to finally achieve a high degree of fusion of defect features.

[0097] S4, based on the fusion feature information processed in step S3, further inputs the region generation network module and the region of interest detection head module, and the step S4 further includes:

[0098] S41, based on the fusion features obtained in step S3, the detection candidate frame is obtained through the region generation network module. The region generation network module detects the area in the feature map where the detection object may exist and forms a candidate frame. In actual operation, a sliding convolution window is used to slide the entire feature map, and anchor frames of different sizes and proportions are used for each window position to calculate the possibility of defects in the anchor frame and calculate its bounding box regression parameters. A large number of defect target candidate frames can be generated. After non-maximum suppression processing and elimination of a large number of overlapping candidate frames, these candidate frames are input into the region of interest detection head module. The loss function of the region generation network module during training is as follows:

[0099]

[0100]

[0101]

[0102] In the formula, p i represents the probability that the i-th anchor box is predicted to be a real defect, When it is a positive sample, it is 1, and when it is a negative sample, it is 0. λ is a constant, t i represents the bounding box regression parameters for predicting the i-th anchor box, represents the parameter of the marked area corresponding to the i-th anchor box, N cls Represents the number of all samples in a mini-batch, N reg Represents the number of anchor box positions in the feature map, ln represents the natural logarithm, Indicates the use of Smooth L1 as the regression loss function;

[0103] In step S42, the candidate box obtained in step S3 is input into the region of interest detection module to further locate and classify the defect targets within the candidate box. The model detection results are compared with the manual labeling results, and the model loss function is calculated. The detection model parameters are continuously iteratively optimized based on the loss value, ultimately forming a target detection model for metal diaphragm surface defects. The region generation network module and the region of interest detection module share the fusion features obtained in step S31.

[0104] S5, continuously photographing the metal diaphragm workpieces actually produced, Figure 5 An embodiment of using multiple cameras with multiple light sources to shoot metal diaphragm workpieces is shown; the images of the same area shot by multiple cameras, such as Figure 6 As shown in the figure, in the metal diaphragm surface defect detection model generated by training according to channel input S2~S4, the defects in the multi-channel image are detected and the locations and types of metal diaphragm surface defects appearing in the image are marked with rectangular boxes; the common defect types on the metal diaphragm surface are as follows Figure 7 shown.

[0105] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A metal diaphragm surface defect detection method based on cross-feature fusion, characterized in that: The following steps are involved: S1: Use multiple cameras and different light sources to collect images of the metal diaphragm surface. Multiple images collected in the same area of ​​the metal diaphragm are grouped together to establish a corresponding relationship. The locations and types of metal diaphragm surface defects in the collected images are manually annotated to form a metal diaphragm surface defect dataset, which is divided into a training set and a validation set. S2, input the training samples in the training set into the visual converter backbone network to extract features to obtain the feature map of the defect image; The visual converter backbone network consists of multiple stages, each of which generates different feature maps. The length and width of the feature maps are halved at each stage, forming a spatially decreasing hierarchical structure. The feature map information of different stages is input into the feature pyramid module. The visual converter backbone network divides the image into multiple window areas for serialization operations. Adjacent windows overlap with each other to preserve the local continuity of the image. S3, based on the different scale feature maps processed in step S2, the feature maps are further input into the improved cross feature pyramid module to perform feature fusion on the feature maps; the improved cross feature pyramid module includes a bottom-up path, a top-down path, a lateral connection path and a cross fusion path, the improved cross feature pyramid module performs feature dimensionality reduction and feature dimensionality increase on the feature maps of different resolutions respectively, and merges the feature maps of corresponding resolutions to generate a merged feature map; the original input picture, the feature maps of different scales generated after step S2, are regarded as the bottom-up path in the improved cross feature pyramid module, and some feature maps are selected in the bottom-up path as the overall input of the improved cross feature pyramid module. The feature maps are respectively subjected to preliminary feature fusion through the lateral connection path and the cross fusion path to obtain a first-level intermediate feature map, wherein the lateral connection path uses a 1×1 convolution layer for connection, and the output dimension conversion is completed while the lateral connection is performed. The cross fusion path uses an improved upsampling module, and fuses the 1×1 convolution layer while using the nearest neighbor interpolation method, so that the upsampling module can better achieve a balance between the dimensions of feature maps of different scales; S4, based on the fusion feature information processed in step S3, further inputs the region generation network module and the region of interest detection head module; S5, continuously photographing the metal diaphragm workpiece in actual production, inputting the images of the same area taken by multiple cameras into the metal diaphragm surface defect detection model trained and generated by S2~S4 according to channels, detecting defects in the multi-channel images and using rectangular frames to mark the locations and types of metal diaphragm surface defects appearing in the images.

2. The metal diaphragm surface defect detection method based on cross feature fusion according to claim 1 is characterized in that: The step S2 further comprises: S21, the pictures in the training set and the corresponding multiple pictures are simultaneously used as input according to different channels, and the annotation information of the input pictures in different channels is subjected to non-maximum suppression in the channel space to eliminate highly repeated annotation information; specifically, the manual labels of the three pictures in the same group are respectively input into the first, second, and third image channels of the defect detection network, that is, non-maximum suppression is performed on the labels between different images at the channel level, and the highly overlapping label information between different images is eliminated, while the image label information in the same image channel is not compared. When performing the non-maximum suppression operation, the IOU of the two defect annotation boxes of the same type needs to be calculated, and the calculation formula is: IOU=Ara(A∩B)Area(A∪B) Where A and B represent two different marked areas respectively. Let the coordinates of the upper left corner and the lower right corner of the rectangular area marked by A be (X 11 , Y 11 )、(X 12 , Y 12 ), the coordinates of the upper left corner and the lower right corner of the rectangular area marked by B are (X 21 , Y 21 )、(X 22 , Y 22 ); The overlapping area is: Area(A∩B)=abs(X 12 -X 21 )×abs(Y 12 -Y 21 ) Union area: Area(A∪B)=abs(X 11 -X 12 )×abs(Y 11 -Y 12 )+abs(X 21 -X 22 )×abs(Y 21 -Y 22 )-abs(X 12 -X 21 )×abs(Y 12 -Y 21 ) Where abs represents the absolute value operation. In actual operation, the image annotation information input from the first image channel is first used as the reference to mark the rectangular box, and the image annotation information input from the second and third image channels is traversed to mark the rectangular box. When the IOU of the marked rectangular box with the reference marked rectangular box is greater than a certain threshold, it is removed. Then, the image annotation information input from the second image channel is used as the reference to mark the rectangular box, and the image annotation information input from the third image channel is traversed to mark the rectangular box. The above operation is repeated; the processed image group is input into the visual converter backbone network; In step S22, the image group processed in step S21 is input into the visual converter backbone network according to the channel to further extract features to obtain a feature map of the defect image. The serialization operation converts the original two-dimensional image into a one-dimensional vector. When the input image has a height of H, a width of W, and a number of channels of C, the serialization operation serializes the input image into N image blocks, and the image block size is: P×P×C Where P is the height and width of the image block. The two-dimensional image is expanded into a one-dimensional vector through serialization operation, where the norm of the one-dimensional vector is: N×P 2 C The one-dimensional vector is input into the converter encoder module. The visual converter backbone network controls the output size of the backbone network feature map through a gradual change strategy of the image block parameter P, where the image block size of the i-th stage is defined as P i , for the input feature map F of the i-th stage i-1 , can be decomposed into multiple image blocks, the number of which is: (H i-1 ×W i-1 ) / P i 2 Each image block is then converted into a one-dimensional vector and projected onto C i Dimension, that is, the output feature map size dimension of the i-th stage is: (H i-1 / P i )×(W i-1 / P i )×C i By flexibly changing the image block size P, the output of feature maps of different scales can be achieved.

3. The metal diaphragm surface defect detection method based on cross feature fusion according to claim 1 is characterized in that: In step S3, the coordinate transformation formula of the nearest neighbor interpolation is as follows: srcX=dstX×(srcW / dstW) srcY=dstY×(srcH / dstH) In the formula, dstX and dstY are the horizontal and vertical coordinates of a pixel in the target image, dstW and dstH are the length and width of the target image, srcW and srcH are the width and height of the original image; (srcX, srcY) is the original image coordinate corresponding to the target image point (dstX, dstY); After completing the initial feature fusion to obtain the first-level intermediate feature map, the feature map of the defect image is further fused through the horizontal connection path and the cross fusion path to obtain the second-level intermediate feature map. The cross fusion path uses an F×F convolution layer with a step size parameter of S and a padding parameter of P to achieve overall downsampling of the first-level intermediate feature map. The calculation formula for the feature map size after overall downsampling is as follows: N=(W-F+2×P) / S+1 Where N represents the output feature map size, W represents the input feature map size, F represents the convolution kernel size, P represents the padding parameter, and S represents the step size parameter; After the downsampling operation, the eigenvalues ​​in the feature map are calculated using the following formula: x=i+s h (m-1)-p h y=j+s w (n-1)-p w Among them, O(i,j) represents the eigenvalue of position (i,j) in the output feature map, I(x,y) represents the eigenvalue of position (x,y) in the input feature map, and K(m,n) represents the weight of position (m,n) in the convolution kernel; h and s w Represents the step size of the convolution kernel in the row and column directions, p h and p w Respectively represent the number of zero padding added in the height and width directions of the input feature map; On the basis of the secondary intermediate feature maps, horizontal connections are performed respectively, and upsampling and feature fusion operations are performed step by step to obtain the output feature maps. Finally, the output feature maps pass through the output convolution layer to achieve the final output.

4. The metal diaphragm surface defect detection method based on cross feature fusion according to claim 1, characterized in that: The step S4 further comprises: S41, based on the fusion features obtained in step S3, a detection candidate frame is obtained through a region generation network module; the region generation network module detects the area in the feature map where the detection object may exist and forms a candidate frame. In actual operation, a sliding convolution window is used to slide the entire feature map, and anchor frames of different sizes and proportions are used for each window position to calculate the possibility of defects in the anchor frame and calculate its bounding box regression parameters. A large number of defect target candidate frames can be generated. After non-maximum suppression processing and elimination of a large number of overlapping candidate frames, these candidate frames are input into the region of interest detection head module. The loss function during training of the region generation network module is as follows: In the formula, p i represents the probability that the i-th anchor box is predicted to be a real defect, When it is a positive sample, it is 1, and when it is a negative sample, it is 0. λ is a constant, t i represents the bounding box regression parameters for predicting the i-th anchor box, represents the parameter of the marked area corresponding to the i-th anchor box, N cls Represents the number of all samples in a mini-batch, N reg Represents the number of anchor box positions in the feature map, ln represents the natural logarithm, Indicates the use of Smooth L1 as the regression loss function; S42, based on the candidate frame obtained in step S3, the defect target in the candidate frame is input into the region of interest detection head module to further locate and classify the defect target, and the model detection result is compared with the manual marking result, and the model loss function is calculated. The detection model parameters are continuously iterated and optimized according to the loss value, and finally a target detection model for surface defects of the metal diaphragm is formed; wherein the region generation network module and the region of interest detection head module share the fusion features obtained in step S3.

Images