A method for generating facility defect images based on a generative adversarial network and image fusion

Abstract

The application discloses a kind of based on generative adversarial network and image fusion facility defect image generation method, S1: the image during normal service of infrastructure is collected;S2: collect limited defect image;S3: use limited defect image to train generative adversarial network, and more style defect image is generated using generative adversarial network;S4: using background image and defect image establishes the method of poisson image fusion;S5: the large number of infrastructure images and defect images are fused, to generate a large number of infrastructure appearance images with defect.The application first uses generative adversarial network to generate a large number of simple background defect images, and combines poisson image fusion technology to fuse simple background defect image with normal service image of infrastructure, to generate a large number of infrastructure defect images with real background.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of infrastructure health monitoring, and particularly relates to a method for generating infrastructure defect images based on a generative adversarial network and image fusion. BACKGROUND

[0002] Defect detection is of great significance in the field of infrastructure health monitoring. Defects in infrastructure can affect road traffic and normal production and life. If these infrastructure defects are not detected in time, it will cause huge national property losses, and may also induce potential traffic and building collapse accidents, causing casualties. In recent years, the use of a large number of infrastructure has accelerated economic development and improved people's living standards. It is an important support for people's production and life. Only by ensuring its safety can people's daily life be carried out smoothly. Therefore, timely detection of infrastructure defects is particularly important. How to accurately and quickly detect road defects is an important problem currently faced.

[0003] Infrastructure defect detection technology has developed rapidly in recent years. Traditional detection methods usually use manual inspection methods, which not only require a large amount of human resource cost, but also have high detection time cost. Subsequently, computer image recognition methods are gradually introduced to improve work efficiency, but they still need to be judged by experts' professional judgment. There is a great subjectivity in the judgment process, and the detection time of the traditional image recognition method is still huge. With the continuous progress of computer image recognition technology, artificial intelligence is gradually applied to the detection task of infrastructure defects. Artificial intelligence technology has strong learning ability and can learn corresponding rules from historical knowledge, and then be used for corresponding recognition tasks. The most critical is the historical knowledge available for learning, or the training (learning) sample of artificial intelligence, because the premise of obtaining an excellent detection model is to provide a large number of learning samples. In actual structures, the number of defects is limited, and the difficulty of data collection is the primary problem restricting the application of artificial intelligence in infrastructure defect detection. The generative adversarial network can generate a large number of data with similar characteristics to the real data, which can expand the original data samples. However, in the actual use process, it is difficult for the generative adversarial network to generate infrastructure defect images with complex real backgrounds. Therefore, how to obtain more complex and real infrastructure defect images is an urgent problem to be solved SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the deficiencies in the prior art, the present application provides a method for generating infrastructure defect images based on a generative adversarial network and image fusion, to solve the problems raised in the background art.

[0006] (II) Technical solutions

[0007] To achieve the above object, the present application provides the following technical solutions.

[0008] A method for generating infrastructure defect images based on a generative adversarial network and image fusion, comprising the following steps:

[0009] S1: Collecting images of infrastructure during normal service;

[0010] S2: Collecting limited defect images (cracks, pits, missing bars, exposed, landslides, etc.);

[0011] S3: Training a generative adversarial network using limited defect images and generating more styles of defect images using the generative adversarial network;

[0012] S4: Establishing a Poisson image fusion method using background images and defect images;

[0013] S5: Fusing a large number of infrastructure images collected with defect images to generate a large number of infrastructure apparent images with defects.

[0014] Preferably, the method is applicable to normal and defect images of various infrastructures (bridges, pavements, slopes, etc.).

[0015] Preferably, the limited defect images are combined with the generative adversarial network technology to generate a large number of defect images.

[0016] Preferably, the Poisson image fusion technology is used to fuse a large number of normal infrastructure images with defect images.

[0017] Preferably, the MATLAB platform is used to build an interactive operation process to realize a large-scale and convenient image fusion technology process.

[0018] Preferably, the images are collected, defect images are generated, a Poisson image fusion method is established, and a large number of infrastructure defect images are generated.

[0019] Compared with the prior art, the method for generating infrastructure defect images based on a generative adversarial network and image fusion has the following beneficial effects:

[0020] 1. The method for generating infrastructure defect images based on a generative adversarial network and image fusion generates a large number of defect images with the same characteristics as real defects using the unsupervised learning algorithm of the generative adversarial network, fuses these defect images with actual infrastructure images, obtains a large number of infrastructure defect images with real complex backgrounds, and uses them as training samples for artificial intelligence recognition technology.

[0021] 2、The infrastructure defect image generation method based on the generative adversarial network and image fusion can significantly improve the data quantity of the infrastructure defect image and has relatively high reliability.

[0022] 3、The infrastructure defect image generation method based on the generative adversarial network and image fusion can simultaneously contribute to the field of automatic identification of infrastructure by artificial intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The bridge, road surface and slope real image schematic diagram obtained by various collection devices of the embodiment of the present application is shown in the figure.

[0024] Figure 2 The limited infrastructure defect image schematic diagram with a simple background collected by the embodiment of the present application is shown in the figure.

[0025] Figure 3 The defect image example schematic diagram generated by the generative adversarial network of the embodiment of the present application is shown in the figure.

[0026] Figure 4 The method implementation flow schematic diagram based on the generative adversarial network and image fusion of the embodiment of the present application is shown in the figure.

[0027] Figure 5 The background image and defect image synthesis effect schematic diagram of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments of the present application.

[0029] The examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0030] Embodiment:

[0031] Please refer to Figures 1-5 The infrastructure defect image generation method based on the generative adversarial network and image fusion provided by the present application includes the following steps:

[0032] S1, collecting images of infrastructure during normal service;

[0033] During the normal service of the infrastructure, most of the images are normal (such as Figure 1), but no matter bridge, pavement or slope, its regular inspection is essential, thus forming a large amount of data, for bridge, bridge inspection vehicle is the most popular collection equipment, which can obtain comprehensive bridge images; for pavement, dashcam is a relatively extensive and low-cost collection equipment, thus a large amount of pavement images can be obtained at minimum cost through dashcam; slope inspection is a heavy work, and the flexibility and convenience of unmanned aerial vehicle enable it to be applied to slope inspection work, thus greatly improving work efficiency, and therefore the slope images collected by unmanned aerial vehicle are adopted in the present application;

[0034] S2, collect limited defect images (cracks, pits, broken bars, etc.) with simple backgrounds;

[0035] Because the defects of infrastructure are relatively few under normal service conditions, and it is difficult to obtain, we screen limited defect images (such as Figure 2 ) from the Internet and existing image sets;

[0036] S3, use limited defect images to train a generative adversarial network, and use the generative adversarial network to generate more styles of defect images;

[0037] The generative adversarial network is a deep learning model that can generate data with similar characteristics to the input real data, which consists of a generator and a discriminator (such as Figure 3 ), Once the generative adversarial network is trained, it can generate a large amount of virtual data similar to real data, and the present project uses MATLAB deep learning resources to design the generator and discriminator of the generative adversarial network, the generator consists of a series of transpose convolution processes, which outputs virtual images with similar characteristics to real training images (for example, bridge surface defects); the discriminator consists of a series of convolution processes, which classifies mixed images (real and virtual); in the ideal state, the generator of the generative adversarial network will generate images that can fool the discriminator, and at this time the images have similar characteristics to real images;

[0038] The present application will use MATLAB deep learning toolbox resources to establish the corresponding network structure of the generative adversarial network (input, output image size, network layer number, convolution kernel number, etc.), use the real bridge surface defect (crack, broken bar, corrosion, etc.) images collected as training samples of the generative adversarial network, and use the training samples to obtain an image generator that can generate a large number of images with similar characteristics to real images, to provide sufficient samples for further establishing a deep learning model of bridge surface defects;

[0039] S4, use background images and defect images to establish a poisson image fusion method;

[0040] In the image fusion task, when the foreground is placed on the background, two points need to be ensured: the main content of the foreground itself is as smooth as possible compared to the background; seamless at the boundary, that is, the pixel values of the foreground and the background at the boundary point position need to keep the boundary consistent, therefore, the present application uses Poisson image fusion technology to establish the following infrastructure defect image generation method;

[0041] Figure 4 Wherein u represents the foreground picture to be synthesized, v is the gradient field of u, S is the background picture, and w is the region covered by the foreground in the target image after merging, then is the boundary of w, and the pixel value representation function of the merged image in w is f, and the pixel value representation function outside w is f*;

[0042] At this time, if the ideal effect of image fusion is to be met, the following two conditions need to be met:

[0043] 1. Smooth transition area of foreground image and background image:

[0044] 2. Boundary consistency:

[0045] Based on the above two limiting conditions, the following infrastructure defect image fusion method (module) is established on the MATLAB platform, including importing images, selecting the position of the defect, image fusion, and exporting the fused image, such as Figure 5 , the background and defect image import buttons are clicked in turn, then the position of the defect is selected through the black cross cursor, after the defect is placed, the next operation window is popped up, and it can be selected to add more defects in the image (add another defect in the image with added defects) or enter the next one, and the fusion step of a large number of images can be realized by repeating the above steps;

[0046] S5, the infrastructure images collected and the defect images are interactively fused, so as to generate a large number of infrastructure apparent images with defects;

[0047] On the basis of S4, the background image and the defect image are interactively imported, and the image fusion process of one defect or multiple defects in the image is completed according to the designed operation process.

[0048] In use, a large number of defect images with the same characteristics as the real defects are generated by using an unsupervised learning algorithm to generate a generative adversarial network, and the defect images are fused with the actual infrastructure images to obtain a large number of infrastructure defect images with real complex backgrounds, which are used as training samples for artificial intelligence recognition technology. This method can significantly improve the data quantity of infrastructure defect images, has relatively high reliability, and at the same time, actively contributes to the field of automatic identification of infrastructure by artificial intelligence.

[0049] The above-mentioned embodiments of the present application need a large amount of historical data to learn to complete the corresponding identification and detection task, and the defect data that can be obtained in the actual infrastructure normal service stage is limited, the generative adversarial network can extract the characteristics of the existing data and automatically generate a large amount of data with similar characteristics to the real data, and then provide sufficient learning samples for the deep learning technology, the actual engineering image is complex, and it is difficult to generate defect images with complex background by using the generative adversarial network, the present application first generates a large number of defect images with simple background by using the generative adversarial network, and combines the poisson image fusion technology to fuse the defect images with simple background and the images of the infrastructure in normal service, thereby generating a large number of infrastructure defect images with real background, and solving the above-mentioned problems.

[0050] Although the embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for generating a defect image based on a generative adversarial network and image fusion, characterized in that, The method comprises the following steps: S1: collecting images during normal service of the infrastructure; specifically, collecting bridge images by a patrol vehicle, collecting road surface images by a vehicle event data recorder, and collecting slope images by a drone; S2: collecting limited defect images, including images of cracks, pits, missing bars, bareness, and landslides; specifically, screening limited defect images from the Internet and existing image sets; An interactive operation process is built on a MATLAB platform to realize a large-scale and convenient image fusion technology process; S3: training a generative adversarial network using the limited defect images, and generating more styles of defect images using the generative adversarial network; S4: establishing a Poisson image fusion method using background images and defect images; S5: fusing the collected large number of infrastructure images and defect images to generate a large number of infrastructure apparent images with defects; wherein, The Poisson image fusion method using background images and defect images in S4 is realized through the following steps: Based on two limiting conditions, a fusion module of infrastructure images and defect images is established on a MATLAB platform, including importing images, selecting a defect placement position, image fusion, and exporting fused images; and the above steps are repeated to realize fusion of a large number of infrastructure images and defect images; The selecting of the defect placement position comprises the following steps: The background image and the defect image are imported in sequence, then a black cross cursor is used to select the defect placement position, after the defect is placed, more defects can be added in the image or the next one can be entered; The two limiting conditions are: (1) foreground image and background image over-region smoothing: (2) boundary consistency: where v is the gradient field of the foreground image to be synthesized, S is the background image, w is the region of the target image covered by the foreground after merging, is the boundary of w, f is the pixel value representation function of the merged image within w, and f* is the pixel value representation function outside w. 2.The method of claim 1, wherein the method comprises: The method is suitable for normal and defect images of various infrastructures. 3.The method of claim 1, wherein: The limited defect images are combined with the generative adversarial network technology to generate a large number of defect images. 4.The method of claim 1, wherein: The Poisson image fusion technology is used to fuse a large number of normal infrastructure images and defect images.

5. The method of claim 1-4, wherein the method is based on a generative adversarial network and image fusion facility defect image generation. A large number of infrastructure defect images are generated by collecting images-generating defect images-establishing a Poisson image fusion method.

Images