Method for enhancing artillery barrel inner wall defect image data

Abstract

The invention discloses an artillery barrel inner wall defect image data enhancement method, relates to the technical field of image processing, and adopts the technical scheme that a steel surface defect data set is combined with an artillery barrel inner wall defect-free image through a Poisson fusion method to generate a vivid artillery barrel inner wall defect image. According to the method, the problem that the actual artillery barrel inner wall defect image is difficult to obtain is effectively solved, and rich data resources are provided for training of a deep learning model. A parallel combination of an attention mechanism module and a full-dimensional dynamic convolution (ODconv) module is introduced into an STYLEGAN2 network architecture, and a novel image generation network structure is constructed. According to the structure, the generator can pay attention to local and global features at the same time, and the quality and diversity of generated images are improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for enhancing image data of inner wall defects of a gun barrel. Background Art

[0002] Defects on the inner wall of a gun barrel have a direct impact on the performance and safety of the gun. These defects may lead to reduced shooting accuracy or even dangerous firing accidents. Traditional defect detection methods, such as ultrasonic testing and optical inspection, are often time-consuming and have limited accuracy. With the development of deep learning technology, new possibilities are provided for automatically and efficiently identifying defects on the inner wall of gun barrels.

[0003] However, the training of deep learning models relies on a large amount of high-quality image data, and the acquisition of defect images of the inner wall of gun barrels faces unique challenges. Field acquisition is not only costly, but also due to the use environment of guns, images are often affected by physical factors such as dust, uneven lighting, and surface reflection, resulting in degraded image quality. In addition, due to safety and production efficiency considerations, the image data of gun barrels containing defects is very limited. Therefore, developing data enhancement techniques that can work under small sample conditions is crucial to improving the performance of defect detection models.

[0004] Current data augmentation methods, such as flipping, rotating, and adding noise, can expand the dataset, but they cannot add new defect types or scene diversity. Generative adversarial networks (GANs) have the potential to generate new samples, but they may encounter mode collapse problems during training, resulting in a lack of diversity in the generated images.

[0005] In summary, how to explore and develop new data enhancement technologies to generate high-quality and diverse defect images, thereby improving the accuracy and efficiency of artillery barrel inner wall defect detection, is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] The technical solution of the present invention to solve the above technical problem is to provide a method for enhancing the image data of the inner wall defects of a gun barrel, comprising the following steps:

[0007] Step 1: Obtain a steel surface defect dataset and preprocess the dataset using histogram equalization and Laplace operator sharpening;

[0008] Step 2: Use the preprocessed image as the target image and the gun barrel inner wall defect-free image as the background image, generate a mask image according to the label information of the target image, perform Poisson fusion on the target image, the mask image and the background image, and generate a gun barrel inner wall defect image;

[0009] Step 3: Classify the generated gun barrel inner wall defect images into four categories: cracks, scratches, burns, and wear, and use Labelme to annotate the images to form a complete gun barrel inner wall defect dataset;

[0010] Step 4: Build the StyleGAN2-ADA network architecture, including the generator and the discriminator, and add a new attention mechanism module before the upsampling layer in the comprehensive network of the generator. The new attention mechanism module is composed of a self-attention mechanism module and a full-dimensional dynamic convolution module in parallel;

[0011] Step 5: Use the steel surface defect dataset to initialize the network in the training weights of StyleGAN2-ADA, and then send the gun barrel inner wall defect dataset to the StyleGAN2-ADA network with set parameters for training;

[0012] Step 6: After the training is completed, the optimal weight file is loaded into the network to generate a single type of gun barrel inner wall defect image agreed upon according to the conditional information.

[0013] Furthermore, in step 4, the generator includes a mapping network and an integrated network, the mapping network includes 8 fully connected layers; the integrated network includes 7 resolution layers, and the resolution layers range from 4*4 modules to 256*256; the 4*4 module includes a convolution layer, a togg layer that converts the feature map into an RGB image, and two intermediate layers; the 8*8 and remaining resolution modules include an upsampling layer, a convolution layer and two intermediate layers; the new attention mechanism module is before the upsampling layer.

[0014] Furthermore, in step 4, the discriminator includes a fromrgb layer, 7 resolution modules from 256*256 to 4*4; each resolution module from 256*256 to 8*8 includes a skip layer for realizing a residual structure, skipping feature maps at different scales, and the residual structure includes a downsampling layer and a convolution layer; wherein, after the last downsampling layer, a multi-normalization layer and a fully connected layer are connected; and the last output layer is used to output the image.

[0015] Furthermore, in step 1, the histogram equalization increases the contrast by broadening the grayscale values ​​of important parts of the image and merging the remaining parts. The specific formula is as follows:

[0016] s = clog(1+r).

[0017] Furthermore, in step 1, the Laplacian operator sharpening further increases or decreases the grayscale of the central pixel by judging the grayscale of the central pixel and the average grayscale of other pixels in the neighborhood. The specific formula is as follows:

[0018]

[0019] Furthermore, in step 2, the Poisson fusion includes:

[0020] The mask image, original image, and background image are input into the network, the fusion position in the background image is determined, and the fusion is performed using the following formula:

[0021]

[0022] Among them, ω is the fusion target, V is its gradient field, S is the background image, Ω is the fusion area, is the fusion boundary. After fusion, the pixels inside Ω are f and the pixels outside are f * .

[0023] Furthermore, in step 5, the training process includes setting the learning rate, the number of batch samples, and the number of thousands of images for iteration, and initializing the network with the training weights of StyleGAN2-ADA using the face dataset.

[0024] The technical solution of the present invention combines the steel surface defect dataset with the defect-free image of the inner wall of the gun barrel through the Poisson fusion method to generate a realistic image of the inner wall defect of the gun barrel. This method effectively solves the problem that it is difficult to obtain the actual inner wall defect image of the gun barrel, and provides rich data resources for the training of deep learning models. The parallel combination of the attention mechanism module and the full-dimensional dynamic convolution (ODconv) module is introduced into STYLEGAN2 to construct a novel image generation network structure. This structure enables the generator to focus on local and global features at the same time, improving the quality and diversity of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0026] Figure 1 This is a flow chart of the steps of the method for enhancing the image data of the inner wall defects of a gun barrel according to the present invention;

[0027] Figure 2 This is a schematic diagram of the self-attention mechanism of the present invention;

[0028] Figure 3 This is a diagram showing the influence of different resolutions on the generated image of the present invention;

[0029] Figure 4 These are example images of defect images with different resolutions according to the present invention;

[0030] Figure 5 It is the FID score graph of the training process of the present invention;

[0031] Figure 6 This is an example of defect image of the enhanced data set of the present invention;

[0032] Figure 7 This is an example diagram of defect image detection of the present invention;

[0033] Figure 8 This is the network architecture diagram of StyleGAN2-ADA of the present invention. DETAILED DESCRIPTION

[0034] The present invention proposes a method for enhancing the data of an inner wall defect image of a gun barrel, aiming to design a method for enhancing the data of an inner wall defect image of a gun barrel based on conditional enhancement and attention mechanism.

[0035] The following will describe the gun barrel inner wall defect image data enhancement method proposed by the present invention in a specific embodiment:

[0036] In the technical solution of this embodiment, Figure 1 As shown, a method for enhancing the image data of the inner wall defect of a gun barrel comprises the following steps:

[0037] Step 1: Obtain a steel surface defect dataset and preprocess the dataset using histogram equalization and Laplace operator sharpening;

[0038] Specifically, the preprocessed image is adjusted to 256*256. Histogram equalization: The crack defect image is subjected to histogram equalization. By broadening the grayscale value of the important part of the image and merging the rest, the contrast is increased to solve the problem that the grayscale of the crack defect image is concentrated in the low brightness range due to insufficient light during the acquisition process; Laplace operator sharpening is an image enhancement method derived from the second-order differential. By judging the grayscale of the central pixel and the average grayscale of other pixels in the neighborhood, the grayscale of the central pixel is further increased or decreased to achieve contrast enhancement in the details of other defective images and complete the sharpening process.

[0039] Step 2: Use the preprocessed image as the target image and the gun barrel inner wall defect-free image as the background image, generate a mask image according to the label information of the target image, perform Poisson fusion on the target image, the mask image and the background image, and generate a gun barrel inner wall defect image;

[0040] Specifically,

[0041] Step 3: Classify the generated gun barrel inner wall defect images into four categories: cracks, scratches, burns, and wear, and use Labelme to annotate the images to form a complete gun barrel inner wall defect dataset;

[0042] Step 4: Build the StyleGAN2-ADA network architecture, including the generator and the discriminator, and add a new attention mechanism module before the upsampling layer in the comprehensive network of the generator. The new attention mechanism module is composed of a self-attention mechanism module and a full-dimensional dynamic convolution module in parallel;

[0043] Step 5: Use the steel surface defect dataset to initialize the network in the training weights of StyleGAN2-ADA, and then send the gun barrel inner wall defect dataset to the StyleGAN2-ADA network with set parameters for training;

[0044] Step 6: After the training is completed, the optimal weight file is loaded into the network to generate a single type of gun barrel inner wall defect image agreed upon according to the conditional information.

[0045] Specifically, Figure 8 As shown in the figure, adding an attention mechanism is an important means to improve the performance of the model. By assigning different weights to each area in the image, the generated model can focus on the key information of the defect and reduce the attention to irrelevant areas, thereby improving the quality of the generated image. Compared with other attention mechanisms, when extracting image features and assigning weights, they are constrained by local connections and sequential calculations, and cannot establish interactions at all positions in the world; the self-attention mechanism can directly integrate the positional relationship between any two pixels or features in the image, is not restricted by local structures, processes spatial information globally and in parallel, and dynamically adjusts the attention weight according to the data itself to strengthen the attention to defect information. The principle is as follows: Figure 2 As shown, first, the extracted feature map x is used by W f , W g and W h The three matrices are transformed into three different feature spaces f, g and h to obtain feature maps f(x), g(x) and h(x); secondly, f(x) is transposed and dot-producted with g(x), and the dot-product result is softmax normalized to obtain the attention feature map β; then β is dot-producted with h(x) to obtain the self-attention feature map ο; finally, the self-attention feature map ο is added to the original convolution feature map x to obtain the final feature map y output.

[0046] The full-dimensional dynamic convolution module is designed to extract local features of the image, but the traditional convolution kernel is limited by fixed parameters, resulting in a limited range of what can be captured. The full-dimensional dynamic convolution module (ODConv) adds an attention mechanism to enable the convolution kernel to dynamically adjust its parameters according to different inputs or latent space vectors, and learns its attention values ​​in parallel along the four dimensions of kernel space, input channel dimension, output channel dimension, and kernel dimension, and applies them to the corresponding convolution kernel. Without increasing the amount of computation, all kernels in the convolution operation can provide the ability to capture more complex structures and rich context clues in the data, thereby improving the feature extraction capability of the network's basic convolution operation. The specific formula is:

[0047]

[0048] Among them, α wi ∈R represents the convolution kernel W i Different attention scalars assigned; α si ∈R k×k Represents the different attention scalars assigned to each convolution kernel in the kernel space; Represents the c of each convolution kernel in Different attention scalars for channel assignment; Represents the c of each convolution kernel out Different attention scalars for channel assignment; and They represent input features and output features respectively.

[0049] Although the self-attention mechanism can pay attention to both global information and local details, self-attention usually calculates attention weights for all positions, which may weaken the network's attention to local features. To solve this problem, this application proposes to combine the self-attention mechanism with ODConv to form a new attention mechanism module in parallel, extracting local information through the ODConv layer and global information through the self-attention layer. It can retain local details and use global information for feature learning, while considering local and overall information to achieve high-quality image generation. The information contrast captured by the improved attention mechanism module is shown in Table 1.

[0050] Table 1 Capture information comparison table:

[0051]

[0052] Furthermore, in step 4, the generator includes a mapping network and an integrated network, the mapping network includes 8 fully connected layers; the integrated network includes 7 resolution layers, and the resolution layers range from 4*4 modules to 256*256; the 4*4 module includes a convolution layer, a togg layer that converts the feature map into an RGB image, and two intermediate layers; the 8*8 and remaining resolution modules include an upsampling layer, a convolution layer and two intermediate layers; the new attention mechanism module is before the upsampling layer.

[0053] Furthermore, in step 4, the discriminator includes a fromrgb layer, 7 resolution modules from 256*256 to 4*4; each resolution module from 256*256 to 8*8 includes a skip layer for realizing a residual structure, skipping feature maps at different scales, and the residual structure includes a downsampling layer and a convolution layer; wherein, after the last downsampling layer, a multi-normalization layer and a fully connected layer are connected; and the last output layer is used to output the image.

[0054] Furthermore, in step 1, the histogram equalization increases the contrast by broadening the grayscale values ​​of important parts of the image and merging the remaining parts. The specific formula is as follows:

[0055] s = clog(1+r).

[0056] Furthermore, in step 1, the Laplacian operator sharpening further increases or decreases the grayscale of the central pixel by judging the grayscale of the central pixel and the average grayscale of other pixels in the neighborhood. The specific formula is as follows:

[0057]

[0058] Furthermore, in step 2, the Poisson fusion includes:

[0059] The mask image, original image, and background image are input into the network, the fusion position in the background image is determined, and the fusion is performed using the following formula:

[0060]

[0061] Among them, ω is the fusion target, V is its gradient field, S is the background image, Ω is the fusion area, is the fusion boundary. After fusion, the pixels inside Ω are f and the pixels outside are f * .

[0062] Furthermore, in step 5, the training process includes setting the learning rate, the number of batch samples, and the number of thousands of images for iteration, and initializing the network with the training weights of StyleGAN2-ADA using the face dataset.

[0063] Experimental analysis:

[0064] In order to verify the enhancement effect of the data enhancement method proposed in this paper on the inner wall defect data of the gun barrel, the generated results at different stages were analyzed.

[0065] First, select images with different resolutions. The images in the gun barrel inner wall defect dataset are adjusted to 128×128, 256×256, and 512×512 resolutions, and sent to the improved network for training. The results are as follows: Figure 3 As shown, it can be seen that as the resolution increases, the FID value gradually decreases. This is because for low-resolution images, the generator can obtain less effective information and cannot fully learn the feature distribution of the original image, resulting in low quality of the generated image.

[0066] Figure 4 The defect images of the inner wall of the gun barrel with different resolution sizes are displayed respectively, so that the differences in the generated images can be observed more directly. Figure 4 The resolutions of (a), (b), and (c) are 128×128, 256×256, and 512×512, respectively. It can be seen that the defect features of 512×512 are more obvious and easier to observe.

[0067] Secondly, we select the generated images at different times. Figure 5 It can be seen that the number of iterations is proportional to the quality of the generated image and inversely proportional to the FID value. At 40 kimg, the FID value is 58.3666, and the generated image has a general shape, but the quality of the generated image is low; with the increase in the number of training times, at 680 kimg, the FID value is 24.0271, the image quality improves, and the final FID value reaches 22.8156, which can clearly observe the various defects generated.

[0068] Finally, when generating images, according to the input defect type labels (class_idx), corresponding defect images (crack (0), scratch (1), wear (2), burn (3)) are generated to achieve a balance in the number of defects of each type. The comparison of the number of defects in the generated images is shown in Table 2.

[0069] Table 2 Comparison of defect label quantity:

[0070]

[0071] Since the average detection accuracy of burn defects before data enhancement is 0.843 (YOLOv7), which is better than other types of defects, and the AP value is significantly improved after a small amount of enhancement, only a small amount of data enhancement is performed on this type of defect; considering the balance problem with the number of labels of other defect types, its number is enhanced to 698; finally, through the data enhancement method proposed in this paper, the number of labels of various types of defects reaches a balanced ratio of 1:1:1:0.7, which helps the detection model learn the characteristics of various defects and improve the generalization ability of the model. Figure 6 This is an example of an image of a gun barrel inner wall defect in the enhanced dataset.

[0072] The YOLO network is commonly used for product defect detection in the industrial field. This application verifies the effectiveness of the proposed data augmentation method based on the YOLO model. The number of training rounds is set to 300, the ratio of the training set, validation set, and test set is 8:1:1, and the evaluation indicators are P, R, and mAP.

[0073] First, the data enhancement method proposed in this application is used to expand a small amount of artillery barrel inner wall defect dataset (1982 images), and compared with the dataset without data enhancement on the YOLOv7 model to verify the effectiveness of data enhancement. The comparison is shown in Table 3.

[0074] Table 3 A small amount of data enhancement comparison table:

[0075]

[0076] It can be seen from Table 3 that after a small amount of data enhancement on the original data set, the accuracy of the gun barrel inner wall defect detection can be effectively improved (4.5%), indicating that as the number of images in the data set increases, the model detection accuracy increases accordingly.

[0077] Secondly, the data enhancement method of this application is compared with other data enhancement methods in the YOLOv5 network to further verify the effectiveness of the data enhancement method of this paper. The results of different data enhancement methods are shown in Table 4:

[0078] Table 4 (a) Comparison of training results of data enhancement methods:

[0079]

[0080] Table 4(b) Comparison of data enhancement method verification results:

[0081]

[0082] As shown in Table 4, by comparing different data enhancement methods, it can be seen that after using the method proposed in this application to enhance the data set of gun barrel inner wall defects, the average detection accuracy (mAP) of the model is 75.5%, which is higher than the average detection accuracy of no data enhancement (55.4%), adding noise (62.2%), rotation (64.4%), translation (62.4%), two composite enhancement methods (60.2%, 63.5%, 62.3%), and Mosaic (72.6%). This is because the data enhancement method proposed in this application not only expands the scale of the data set, but also uses the improved StyleGAN2-ADA to generate gun barrel inner wall defect images with the same distribution as the image features of the original gun barrel inner wall defect data set, which effectively expands the diversity of the original samples, thereby improving the generalization ability of the model.

[0083] Figure 7 This is an example of the target detection results of a randomly selected artillery barrel inner wall defect image. It can be seen that the noise-adding method can effectively improve the detection rate of scratches (b), which is equivalent to the present application, but it will cause false detection of crack defects; translation (c) will miss the detection of wear-type defects; rotation (d) has a certain improvement in the detection accuracy of wear and burn-type defects; and after enhancement using the data enhancement method (e) proposed in this article, the false detection rate is effectively reduced, and various defects can be correctly detected.

[0084] After using the data enhancement method in this paper, the comparison of the accuracy improvement of various defects is shown in Table 5.

[0085] Table 5 Comparison of data enhancement methods in this paper:

[0086]

[0087] As shown in Table 5, by comparing the P and AP of various defects, it can be seen that after using the method proposed in this paper to enhance the data set of the inner wall defects of the gun barrel, the P and AP of various defects are improved, especially the crack and burn defects. The crack P is increased by 28.7%, the AP is increased by 34.5%, the burn AP is increased by 22.4%, and the wear P is also greatly improved (18.4%); the overall mAP of the model is improved by 20.1%.

[0088] The present application proposes a method for data enhancement of inner wall defects of artillery barrels based on conditional enhancement and attention mechanism. Experiments have shown that the data enhancement method proposed in the present application is superior to other data enhancement methods, can improve the quality of generated images, and improve the detection accuracy of the detection model. After data enhancement by this method, the FID value is reduced by 12.6193 compared with the original network, and the detection accuracy reaches 75.5%, which is higher than the artillery barrel inner wall defect detection method without data enhancement.

[0089] Using the data enhancement method proposed in this paper, the detection model can accurately detect defects on the inner wall of the artillery barrel. By detecting the shape and size of the defects, the severity of the artillery damage can be judged, and relevant measures can be taken in time, which has positive significance for the use of artillery.

[0090] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for enhancing the image data of the inner wall defects of a gun barrel, characterized in that: The following steps are involved: Step 1: Obtain a steel surface defect dataset and preprocess the dataset using histogram equalization and Laplace operator sharpening; Step 2: Use the preprocessed image as the target image and the gun barrel inner wall defect-free image as the background image, generate a mask image according to the label information of the target image, perform Poisson fusion on the target image, the mask image and the background image, and generate a gun barrel inner wall defect image; Step 3: Classify the generated gun barrel inner wall defect images into four categories: cracks, scratches, burns, and wear, and use Labelme to annotate the images to form a complete gun barrel inner wall defect dataset; Step 4: Build the StyleGAN2-ADA network architecture, including the generator and the discriminator, and add a new attention mechanism module before the upsampling layer in the comprehensive network of the generator. The new attention mechanism module is composed of a self-attention mechanism module and a full-dimensional dynamic convolution module in parallel; Step 5: Use the steel surface defect dataset to initialize the network in the training weights of StyleGAN2-ADA, and then send the gun barrel inner wall defect dataset to the StyleGAN2-ADA network with set parameters for training; Step 6: After the training is completed, the optimal weight file is loaded into the network to generate a single type of gun barrel inner wall defect image agreed upon according to the conditional information.

2. The method for enhancing the image data of the inner wall defect of a gun barrel according to claim 1, characterized in that: In step 4, the generator includes a mapping network and an integrated network, the mapping network includes 8 fully connected layers; the integrated network includes 7 resolution layers, and the resolution layers range from 4*4 modules to 256*256; the 4*4 module includes a convolution layer, a togg layer that converts the feature map into an RGB image, and two intermediate layers; the 8*8 and remaining resolution modules include an upsampling layer, a convolution layer and two intermediate layers; the new attention mechanism module is before the upsampling layer.

3. The method for enhancing the image data of the inner wall defect of a gun barrel according to claim 1, characterized in that: In step 4, the discriminator includes a fromrgb layer, 7 resolution modules from 256*256 to 4*4; each resolution module from 256*256 to 8*8 includes a skip layer for realizing a residual structure, skipping feature maps at different scales, and the residual structure includes a downsampling layer and a convolution layer; wherein, after the last downsampling layer, a multi-normalization layer and a fully connected layer are connected; and the last output layer is used to output the image.

4. The method for enhancing the image data of the inner wall defect of a gun barrel according to claim 1, characterized in that: In step 1, the histogram equalization increases the contrast by broadening the grayscale values ​​of the important parts of the image and merging the rest. The specific formula is as follows: s = clog(1+r); Among them, c is the adjustment constant, r is each pixel of the image before transformation, and s is the output pixel after transformation.

5. The method for enhancing the image data of the inner wall defect of a gun barrel according to claim 1, characterized in that: In step 1, the Laplacian operator sharpening further increases or decreases the grayscale of the central pixel by judging the grayscale of the central pixel and the average grayscale of other pixels in the neighborhood. The specific formula is as follows: in, It represents the sum of the second-order partial derivatives of the gray value of the central pixel f, that is, the change characteristics of the central gray value and the gray value of the pixels in the (x, y) neighborhood.

6. The method for enhancing the image data of the inner wall defect of a gun barrel according to claim 5, characterized in that: In step 2, the Poisson fusion includes: The mask image, original image, and background image are input into the network, the fusion position in the background image is determined, and the fusion is performed using the following formula: Among them, ω is the fusion target, V is its gradient field, S is the background image, Ω is the fusion area, is the fusion boundary. After fusion, the pixels inside Ω are f and the pixels outside are f * .

7. The method for enhancing the image data of the inner wall defect of a gun barrel according to claim 1, characterized in that: In step 5, the training process includes setting the learning rate, the number of batch samples, and the number of thousands of images for iteration, and initializing the network using the training weights of the face dataset in StyleGAN2-ADA.

Images