GAN-based ship beam surface defect detection data augmentation method

Abstract

The embodiment of the invention provides a GAN-based ship beam surface defect detection data augmentation method, which comprises the following steps: acquiring a part of ship beam surface defect data set, and dividing the ship beam surface defect data set into a training set and a test set; a conditional generative adversarial network is adopted, an object generation network is constructed, and variables adopted by the conditional generative adversarial network are type labels of the ship beam surface defects; constructing an image conversion network, combining the defect image generated by the object generation network with the defect-free real image to generate a defective picture, and enabling the generated image to be similar to the target image; training an object generation network, generating a part of defect objects by using the constructed object generation network, and training an image conversion network by using the generated defect objects and real defect objects in the ship beam surface defect data set; a random noise and defect category label is input into a trained ship beam surface defect generation model, and a corresponding ship beam surface defect picture with the category label is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image data augmentation, and in particular to a GAN-based ship beam surface defect detection data augmentation method. Background Art

[0002] In the field of shipbuilding and maintenance, ship beams are key structural support components, and the detection of surface defects is crucial. Traditional methods for detecting surface defects in ship beams mainly rely on manual visual inspection, which is time-consuming and labor-intensive, and is easily affected by subjective factors, resulting in limited consistency and accuracy of the detection results. With the development of artificial intelligence and computer vision technology, automated defect detection methods have gradually become a hot topic of research. Generative adversarial networks (GAN), as a powerful deep learning model, have demonstrated excellent performance in many fields such as image generation, style transfer, and data enhancement. Since ship beams are key load-bearing components in the hull structure, their manufacturing and inspection standards are usually very strict, so their defect probability is low. However, during the long-term use of ship beams, ship beams may have defects such as corrosion, structural cracks, and welding defects. These defects may appear in places with uncertain locations, and the characteristics of some defects are not obvious, and insufficient defect images can be collected. In this way, the trained model is prone to missed detection and loss during the detection process. Summary of the invention

[0003] In view of the above problems existing in the prior art, an embodiment of the present invention provides a GAN-based ship beam surface defect detection data augmentation method, which uses a small amount of defect detection data set for training, converts defect-free pictures into realistic defect images, and marks the locations of defects, which can save the work of manual labeling. At the same time, it can control the types of generated defects and flexibly perform defect data augmentation, which can well solve the problem of too few defect data sets in the ship beam surface defect detection task.

[0004] The embodiment of the present invention provides a GAN-based ship beam surface defect detection data augmentation method, comprising:

[0005] Obtain a partial data set of ship beam surface defects and divide it into a training set and a test set;

[0006] A conditional generative adversarial network is used to construct an object generation network, wherein the variable used by the conditional generative adversarial network is a type label of the surface defect of the ship beam;

[0007] Construct an image conversion network to merge the defective image generated by the object generation network with the real image without defects to generate a defective picture, so that the generated image is similar to the target image;

[0008] Training the object generation network, using the constructed object generation network to generate a part of defect objects, and using the generated defect objects and the real defect objects in the ship beam surface defect dataset to train the image conversion network;

[0009] By inputting a random noise and a defect category label into the trained ship beam surface defect generation model, the corresponding ship beam surface defect image with the same category label is generated.

[0010] In some embodiments of the present invention, the beam surface defect dataset includes defect images actually collected from the hull beam;

[0011] The ship beam surface defect dataset includes three types of defects: corrosion defects, structural cracks and welding defects.

[0012] In some embodiments of the present invention, the conditional generative adversarial network is used to construct an object generation network, and the variable used by the conditional generative adversarial network is a type label of the surface defect of the ship beam, including:

[0013] The type label of the ship beam surface defect is used as a conditional variable and input into the generator and discriminator at the same time. The network is generated using the real target defect training objects cropped from the ship beam surface defect dataset.

[0014] In some embodiments of the present invention, in the process of training the object generation network, the training of the generator and the discriminator is specifically to alternately train the generator and the discriminator.

[0015] In some embodiments of the present invention, when constructing an image conversion network, the loss function of CGAN is used, which is as follows:

[0016] Set the conditional variable y, merge the conditional variable y with the input data x and noise z respectively, and use them as the input of the generator G and the discriminator D. Therefore, the loss function of CGAN is:

[0017] V(D,G)=E x～Pdata(x) [log D(x,y)]+E z～Pz(z) [log(1-D(G(z,y)))]

[0018] Among them, z is the noise randomly drawn from the normal distribution, and y is the category label, specifically the category label of the defect.

[0019] In some embodiments of the present invention, merging the defective image generated by the object generation network with the real image without defects to generate the defective image so that the generated image is similar to the target image includes:

[0020] Build an image translation network generator using an encoder-decoder architecture, including a residual network block;

[0021] The constructed image conversion network discriminator includes a global discriminator and a local discriminator, and the two discriminators are used to perform the image conversion and natural mixing tasks of the image conversion network.

[0022] Compared with the prior art, the GAN-based ship beam surface defect detection data augmentation method provided in the embodiment of the present invention generates ship beam surface images with various types of defects to increase the diversity of training data and improve the detection performance of the defect detection model. The model of the present invention is divided into two networks: an object generation network and an image conversion network. The object generation network generates defect images within the bounding box of the data set, and the image conversion network merges these defects with defect-free images to generate defect images. The defect images generated by this scheme can improve the defect detection capability of the defect detection model.

[0023] The present invention adopts a divide-and-conquer approach to divide the model into an object generation network and an image conversion network, and uses the image generated by the object generation network as the input of the image conversion network. This can improve the stability of the image conversion network during training and make the generated defect objects closer to real defect images.

[0024] The object generation network of the present invention adopts a conditional adversarial generation network, and the type label of the surface defect of the ship beam is input into the object generation network as a conditional variable, which can control the defect type generated by the object generation network, thereby avoiding data imbalance when generating defect images.

[0025] The image conversion network of the present invention adopts a global discriminator and a local discriminator. The global discriminator evaluates small blocks of the image and checks whether these parts are similar to the target image. It can effectively capture the overall layout and style of the image and evaluate the authenticity of the overall image. The local discriminator determines whether the defective object at the corresponding position is real, and can more carefully judge the quality of the generated defects, such as texture and details. Combining these two discriminators, the network can generate results with rich details while ensuring the overall consistency of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The overall architecture diagram of the GAN-based ship beam surface defect detection data augmentation method provided in an embodiment of the present invention;

[0027] Figure 2 A diagram of the object generation network system structure in the GAN-based ship beam surface defect detection data augmentation method provided in an embodiment of the present invention;

[0028] Figure 3A diagram of the architecture of an image conversion network generator in a GAN-based ship beam surface defect detection data augmentation method provided in an embodiment of the present invention;

[0029] Figure 4 A diagram of the architecture of the image conversion network discriminator in the GAN-based ship beam surface defect detection data augmentation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0032] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0033] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0034] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0035] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to determine the true intent based on the user's historical operations and to avoid unnecessary or redundant details that make the present application unclear. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but are merely used as the basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0036] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0037] The embodiment of the present invention provides a data augmentation method for ship beam surface defect detection based on GAN, which can generate a ship beam surface defect generation model. The scheme uses a small amount of ship beam surface defect data set for training, and converts the image into a picture similar to the original picture while generating the specified defect. The scheme divides the model into two networks: an object generation network and an image conversion network. The object generation network is used to generate the target defect, and the image conversion network combines the generated defect with the defect-free image, converts it into a defective image, and records the defect location.

[0038] The GAN-based ship beam surface defect detection data augmentation method specifically includes:

[0039] A small amount of ship beam surface defect datasets are obtained and divided into a training set and a test set, wherein the ship beam surface defect dataset used is defect images actually collected from the ship hull beam;

[0040] A conditional generative adversarial network is used to construct an object generation network. The object generation network adopts a conditional generative adversarial network (CGAN). In this embodiment, the conditional generative adversarial network CGAN adopted by the object generation network is specifically that a conditional variable is added on the basis of the traditional GAN, and the type label of the ship beam surface defect is input into the generator and the discriminator as a conditional variable at the same time, and the real target defect cut out from the ship beam surface defect dataset is used to train the object generation network;

[0041] An image conversion network is constructed to merge the defective image generated by the object generation network with the defect-free real image to generate a defective picture, so that the generated image is similar to the target image. In order to enable the GAN model to perform this complex task, the constructed image conversion network generator adopts an encoder-decoder architecture, which includes a residual network (ResNet) block; the constructed image conversion network discriminator includes a global discriminator and a local discriminator, and the two discriminators are used to perform the image conversion and natural mixing tasks of the image conversion network.

[0042] Train the image conversion network, use the constructed object generation network to generate a part of defect objects, and use the generated defect objects and the real defect objects in the ship beam surface defect dataset to train the image conversion network together, so that the image conversion network can learn more diverse and complex defect features.

[0043] By using the trained ship beam surface defect generation model and inputting a random noise and defect category label, a ship beam surface defect image with the same category label can be generated.

[0044] In order to facilitate a detailed understanding of the above technical solution, the following is a detailed description with reference to the accompanying drawings and specific examples, as follows:

[0045] like Figure 1 As shown in the figure, it is the overall architecture diagram of the data augmentation method for ship beam surface defect detection based on GAN. The specific steps can be described as follows:

[0046] 1. Obtain a small amount of ship beam surface defect data set, specifically,

[0047] In this embodiment, the ship beam surface defect dataset used is mainly defect images actually collected from the hull beam. After screening, nearly 1,000 images were selected and divided into training set and test set in a ratio of 6:4. The dataset includes three types of defects: corrosion, structural cracks, and welding defects.

[0048] 2. Build an object generation network. Specifically,

[0049] The object generation network uses a conditional generative adversarial network (CGAN). CGAN adds conditional variables to the traditional GAN, which can be any type of auxiliary information, such as class labels, partial data, or any other type of data. In CGAN, these conditional variables are input into both the generator and the discriminator. In this embodiment, the conditional variable here is the type label of the surface defect of the ship beam, which can control the defect type generated by the generation network and avoid data imbalance.

[0050] The object generation network architecture can be found in the attached Figure 2 , the generator accepts a random noise z and a defect type label y as input. The random noise z is randomly sampled from a normal distribution, and the defect category label y is the label corresponding to three defects (corrosion, structural cracks, and welding defects). The input of the discriminator also consists of two parts, one is the real data x or the data generated by the generator, and the other is the conditional variable y corresponding to these data.

[0051] 3. Train (build) the object generation network. Specifically,

[0052] In this embodiment, the object generation network is trained using real target defects cut from the ship beam surface defect dataset. During the training of the object generation network, the generator and the discriminator are trained alternately. The discriminator tries to get better at each iteration to distinguish between true and false data, while the generator tries to deceive the discriminator and generate more realistic data. Through this adversarial training, the generator learns to generate more and more realistic data.

[0053] The loss function of CGAN used in the object generation network takes into account the conditional variable y, and combines the conditional variable y with the input data x and noise z respectively as the input of the generator G and the discriminator D. Therefore, the loss function of CGAN is:

[0054] V(D,G)=E x～Pdata(x) [log D(x,y)]+E z～Pz(z) [log(1-D(G(z,y)))]

[0055] Where z is noise randomly drawn from a normal distribution, and y is a class label, which is the class label of the defect in this invention. For CGAN, the loss function contains conditional information (such as class labels), which allows the model to consider this additional information during the generation process, thereby producing outputs with more specific characteristics.

[0056] 4. Build an image conversion network.

[0057] The image conversion network merges the defective image generated by the object generation network with the real image without defects to generate a defective picture, while making the generated image similar to the target image. In order to enable the GAN model to perform this complex task, the constructed image conversion network generator adopts an encoder-decoder architecture, which contains a residual network (ResNet) block; the constructed image conversion network discriminator includes a global discriminator and a local discriminator, and uses two discriminators to perform the image conversion and natural mixing tasks of the image conversion network.

[0058] 4.1 Build the image conversion network generator.

[0059] The image translation network generator architecture can be found in the attached Figure 3 The image conversion network generator adopts an encoder-decoder architecture, which consists of two downsampling layers, 11 residual network (ResNet) blocks and two upsampling layers. The generator uses a bounding box mask to indicate the location of defect insertion, adjusts the defect image G(z,y) generated by the object generation network to the mask size, and places the defect image G(z,y) at the mask position, and then merges it with the defect-free real image and feeds it into the network. The generator converts it into a defective image, and makes the generated image similar to the input real image background, while recording the location of the defect to avoid the trouble of subsequent manual labeling.

[0060] 4.2 Construct the image conversion network discriminator.

[0061] The image translation network discriminator architecture can be found in the attached Figure 4 , the image translation network discriminator includes a global discriminator and a local discriminator Image translation and natural mixture tasks of the image translation network are performed using two discriminators.

[0062] Global Discriminator Responsible for evaluating the images generated by the image conversion network generator, its structure is based on the PatchGAN discriminator. Unlike evaluating the entire image, the PatchGAN discriminator evaluates small patches of the image to check whether these parts are similar to the target image, which can effectively capture the overall layout and style of the image and evaluate the authenticity of the overall image. The result of this evaluation is used to form an adversarial loss, which is an indicator used during training to guide the generator to improve its generated images to make them more realistic and closer to the target sample.

[0063] Local Discriminator Responsible for judging whether the defect object at the corresponding position is real, helping to capture the detailed information of the image, and judging whether the defect object can be obtained by cropping and resizing operations applied to the generated image. This can judge the quality of the generated defects more finely, such as texture and details. Combining these two discriminators, the network can generate results with rich details while ensuring the overall consistency of the image. This method allows the network to generate high-quality overall images without sacrificing local details.

[0064] 5. Train the image conversion network.

[0065] A part of defect objects is generated by the trained object generation network, and the image conversion network loss function is constructed. The generated defect objects and the real defect objects in the ship beam surface defect dataset are used to train the image conversion network, so that the image conversion network can learn more diverse and complex defect features.

[0066] The loss function L of the image conversion network tr By fighting against loss and background loss It consists of two parts.

[0067] Adversarial loss is used to help the generator learn the mapping from the defect-free image c to the target image T. It consists of two parts: the loss of the global discriminator and the loss of the local discriminator. Global Discriminator Evaluate the authenticity of the entire generated image, while the local discriminator Evaluate the authenticity of specific defect areas in the generated image. These two evaluation results are combined to form the adversarial loss As shown below. It guides the generator to produce more realistic images by penalizing images that can be identified as fake by the discriminator. This adversarial loss not only encourages the generator to generate images that are globally similar to the target domain, but also ensures that the local details of the image are also authentic.

[0068]

[0069] Among them, i t Indicates a real defective picture; G tr (i p ,i c ) represents the defective picture generated by the generator, where i p represents the image generated by the image generation network and the image after the mask is combined, i c represents a real image without defects; R(i t ) represents the defect image cropped from the real defect image; R(G tr (i p ,i c )) represents the defect image at the corresponding position of the image generated by the generator.

[0070] The background loss is used to find the input image i c And the generated image G tr (i p ,i c ) between the areas outside the bounding box mask. Since the generator G tr The encoder-decoder structure is used to first compress the image into a low-dimensional representation and then restore it. The advantage of this design is that the generated image structure is relatively free. However, this also means that the fidelity of the image is reduced. Therefore, in the generated image, the edge components of the image are blurred, resulting in obvious changes in hue while reducing color variance.

[0071] Background loss is used to optimize the consistency between the generated image and the input image in non-target areas. It calculates the input image i c And generate image G tr (i p ,i c ) except mask m b The pixel-level L1 distance outside the region can reduce the focus on the masked area (such as the defect area), thereby improving the quality of other parts of the image. Background loss helps the stable training of the generator and can generate clearer images. The background loss function is as follows:

[0072]

[0073] Among them, i c represents the input image; G tr (i p ,i c ) represents the generated image; m b is the mask, and (1-m b ) represents the area outside the mask. Through this loss function, the generator can better keep the background stable and coherent when generating objects.

[0074] Finally, the complete loss function of the image conversion network is as follows:

[0075]

[0076] Among them, λ 1 is the intensity coefficient of the background loss function, which controls the weight of the background loss in the entire loss function. 1 It will increase the penalty for background loss, making the model pay more attention to maintaining the accuracy and consistency of the background in the generated image. 1 The value of can affect the degree to which the model handles the background when generating images, making it more in line with specific requirements or constraints.

[0077] 6. Use the trained ship beam surface defect generation model, input a random noise and defect category label, and generate a ship beam surface defect image with the same category label.

[0078] The object generation network receives a random noise z of a fixed dimension (taking 128 dimensions as an example) and a 3D category label y, and generates a defect object G of 128×128 pixels. obj The beam surface defect generation model samples the bounding box mask m from a uniform distribution in the beam workpiece area. b , and use m b and G obj (z,y) to create an object i p . The resized object i p With defect-free images c Passed together to the generator G of the image conversion network tr . With a flawless image i c Make the background to create the generated image G tr (i p ,i c ). The image conversion network uses 1280×1280 pixel images as the input and output of the generator image conversion network. After defect image generation for the object detection dataset, the mask is converted into a bounding box, eliminating the trouble of manual annotation.

[0079] It can be seen from the above technical scheme that the GAN-based ship beam surface defect detection data augmentation method provided by the above embodiment of the present invention generates ship beam surface images with various types of defects to increase the diversity of training data and improve the detection performance of the defect detection model. The model of the present invention is divided into two networks: an object generation network and an image conversion network. The object generation network generates defect images within the bounding box of the data set, and the image conversion network merges these defects with defect-free images to generate defect images. The defect images generated by this scheme can improve the defect detection capability of the defect detection model.

[0080] The present invention adopts a divide-and-conquer approach to divide the model into an object generation network and an image conversion network, and uses the image generated by the object generation network as the input of the image conversion network. This can improve the stability of the image conversion network during training and make the generated defect objects closer to real defect images.

[0081] The object generation network of the present invention adopts a conditional adversarial generation network, and the type label of the surface defect of the ship beam is input into the object generation network as a conditional variable, which can control the defect type generated by the object generation network, thereby avoiding data imbalance when generating defect images.

[0082] The image conversion network of the present invention adopts a global discriminator and a local discriminator. The global discriminator evaluates small blocks of the image and checks whether these parts are similar to the target image. It can effectively capture the overall layout and style of the image and evaluate the authenticity of the overall image. The local discriminator determines whether the defective object at the corresponding position is real, and can more carefully judge the quality of the generated defects, such as texture and details. Combining these two discriminators, the network can generate results with rich details while ensuring the overall consistency of the image.

[0083] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A GAN-based data augmentation method for ship beam surface defect detection, characterized in that: include: Obtain a partial data set of ship beam surface defects and divide it into a training set and a test set; A conditional generative adversarial network is used to construct an object generation network, wherein the variable used by the conditional generative adversarial network is a type label of the surface defect of the ship beam; Construct an image conversion network to merge the defective image generated by the object generation network with the real image without defects to generate a defective picture, so that the generated image is similar to the target image; Training the object generation network, using the constructed object generation network to generate a part of defect objects, and using the generated defect objects and the real defect objects in the ship beam surface defect dataset to train the image conversion network; By inputting a random noise and a defect category label into the trained ship beam surface defect generation model, the corresponding ship beam surface defect image with the same category label is generated.

2. The GAN-based ship beam surface defect detection data augmentation method according to claim 1 is characterized in that: The beam surface defect data set includes defect images actually collected from the hull beam; The ship beam surface defect dataset includes three types of defects: corrosion defects, structural cracks and welding defects.

3. The GAN-based ship beam surface defect detection data augmentation method according to claim 2 is characterized in that: The conditional generative adversarial network is used to construct an object generation network, and the variables used in the conditional generative adversarial network are type labels of surface defects of ship beams, including: The type label of the ship beam surface defect is used as a conditional variable and input into the generator and discriminator at the same time. The network is generated using the real target defect training objects cropped from the ship beam surface defect dataset.

4. The GAN-based ship beam surface defect detection data augmentation method according to claim 3 is characterized in that: In the process of training the object generation network, the training of the generator and the discriminator is specifically to alternately train the generator and the discriminator.

5. The GAN-based ship beam surface defect detection data augmentation method according to claim 4 is characterized in that: When constructing the image conversion network, the loss function of CGAN is used, as follows: Set the conditional variable y, merge the conditional variable y with the input data x and noise z respectively, and use them as the input of the generator G and the discriminator D. Therefore, the loss function of CGAN is: V(D,G)E x～Pdata(x) [log D(x,y)]+E z～pz(z) [log(1-D(G(z,y)))] Among them, z is the noise randomly drawn from the normal distribution, and y is the category label, specifically the category label of the defect.

6. The GAN-based data augmentation method for ship beam surface defect detection according to claim 5 is characterized in that: The defect image generated by the object generation network is combined with a real image without defects to generate a defective image, so that the generated image is similar to the target image, including: Build an image translation network generator using an encoder-decoder architecture, including a residual network block; The constructed image conversion network discriminator includes a global discriminator and a local discriminator, and the two discriminators are used to perform the image conversion and natural mixing tasks of the image conversion network.

Images