A method, apparatus, equipment and storage medium for dam surface image restoration

Abstract

This invention relates to the field of image recognition technology, and discloses a method, apparatus, device, and storage medium for dam surface image restoration. The invention first constructs an initial generative adversarial network (GAN) comprising one generator and two discriminators. Further, by fusing visible light and sonar images into the initial GAN ​​for game-theoretic interaction and reaching an equilibrium point, an optimized target GAN is obtained, solving the problems of convergence difficulty and instability in GANs. Finally, the optimized target GAN is used to restore the dam surface image, recovering the parts of the dam surface obscured by debris and generating a complete, high-quality dam surface image, providing support for dam maintenance.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method, apparatus, equipment, and storage medium for restoring images of dam surfaces. Background Technology

[0002] Currently, underwater image acquisition of dam surfaces is often incomplete and unclear due to environmental factors such as obstruction by weeds and other debris. Genetic Adversarial Networks (GANs) can be used to process and restore these images.

[0003] GAN consists of two important parts: the generator (G) and the discriminator (D).

[0004] Generator: Data is generated by machine with the aim of "fooling" the discriminator as much as possible. The generated data is denoted as G(z).

[0005] Discriminator: Determines whether the data is real or generated by the "generator," aiming to identify "fake data" created by the "generator" as much as possible. Its input parameter is x, representing the data, and the output D(x) represents the probability that x is real data. If the output is 1, it means that the data is 100% real, while if the output is 0, it means that the data cannot be real.

[0006] G and D constitute a dynamic adversarial (or game-like) process. As training (adversarial) progresses, the data generated by G becomes increasingly closer to real data, while D's ability to distinguish real data improves. Ideally, G can generate data that is sufficiently indistinguishable from real data; however, D struggles to determine whether the data generated by the generator is genuine, hence D(G(z)) = 0.5. After training, a generative model G is obtained, which can be used to generate indistinguishable data.

[0007] However, in existing technologies, the GAN (Generator-Discriminator Adversarial Network) model is difficult to converge and is unstable. Good synchronization between the generator and discriminator is required, but in actual training, D (Generator) easily converges while G (Discriminator) diverges. Summary of the Invention

[0008] In view of this, the present invention provides a method, apparatus, device and storage medium for dam surface image restoration, in order to solve the problems of GAN adversarial neural network model being difficult to converge and unstable in the prior art.

[0009] In a first aspect, the present invention provides a method for restoring images of a dam surface, the method comprising:

[0010] A set of historical visible light images and historical sonar images of the dam surface are acquired. An initial generative adversarial network (GAN) is constructed, comprising an initial generator, a first initial discriminator, and a second initial discriminator. The initial generator, the first initial discriminator, and the second initial discriminator are iteratively trained using the historical visible light images and historical sonar images to obtain a target generator, a first target discriminator, and a second target discriminator. Based on the target generator, the first target discriminator, and the second target discriminator, a target GAN is determined. A set of visible light images and sonar images of the dam surface to be repaired are acquired and input into the target GAN for processing to obtain images of the repaired dam surface.

[0011] The dam surface image restoration method provided by this invention first constructs an initial generative adversarial network (GAN) comprising one generator and two discriminators. Then, by fusing visible light and sonar images into the initial GAN ​​and engaging in a game-like process to reach an equilibrium point, an optimized target GAN is obtained, solving the problems of convergence difficulty and instability in GANs. Finally, the optimized target GAN is used to restore the dam surface image, recovering the parts of the dam surface obscured by debris and generating a complete, high-quality dam surface image, thus providing support for dam maintenance.

[0012] In one optional implementation, the initial generator, the first initial discriminator, and the second initial discriminator are iteratively trained using historical visible light image sets and historical sonar image sets to obtain a target generator, a first target discriminator, and a second target discriminator, including:

[0013] Historical visible light image sets and historical sonar image sets are input into the initial generator to obtain a fused image set; the fused image set and historical visible light image sets are input into the first initial discriminator, and the fused image set and historical sonar image sets are input into the second initial discriminator; by continuously performing adversarial training on the initial generator, the first initial discriminator and the second initial discriminator, a target generator, a first target discriminator and a second target discriminator are obtained.

[0014] The dam surface image restoration method provided by this invention firstly fuses historical visible light image sets and historical sonar image sets using an initial generator to obtain a fused image set. Secondly, the fused image set and the historical visible light image set are input into a first initial discriminator to distinguish the visible light images. Then, the fused image set and the historical sonar image set are input into a second initial discriminator to distinguish the sonar images. Finally, the initial generator, the first initial discriminator, and the second initial discriminator are continuously subjected to adversarial training until a trained target generator, a first target discriminator, and a second target discriminator are obtained. This solves the problems of convergence difficulty and instability of generative adversarial networks and provides support for subsequent restoration of dam surface images.

[0015] In one optional implementation, a historical visible light image set and a historical sonar image set are input into an initial generator to obtain a fused image set, including:

[0016] Feature extraction is performed on historical visible light image sets and historical sonar image sets respectively to obtain visible light image feature sets and sonar image feature sets; the visible light image feature sets and sonar image feature sets are fused to obtain fused image feature sets; the fused image feature sets are deconvolutionally reconstructed to obtain fused image sets.

[0017] The dam surface image restoration method provided by this invention first extracts features from historical visible light image sets and historical sonar image sets, respectively. Second, the extracted visible light image feature sets and sonar image feature sets are fused. Finally, the fused image feature set is deconvolutionally reconstructed to obtain the final fused image set, which supports the training of the subsequent initial generator.

[0018] In one optional implementation, the target generator, the first target discriminator, and the second target discriminator are obtained by continuously performing adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator, including:

[0019] The initial generator, the first initial discriminator, and the second initial discriminator are subjected to adversarial training, and the first probability output by the initial generator, the second probability output by the first initial discriminator, and the third probability output by the second initial discriminator are obtained. The first difference value between the first probability and the second probability, and the second difference value between the first probability and the third probability are obtained. It is determined whether the first difference value and the second difference value reach the preset target. If the first difference value and the second difference value do not reach the preset target, the adversarial training of the initial generator, the first initial discriminator, and the second initial discriminator is iterated repeatedly until the first difference value and the second difference value reach the preset target, at which point the training stops, and the target generator, the first target discriminator, and the second target discriminator are obtained.

[0020] The dam surface image restoration method provided by the present invention can further determine whether to continue adversarial training by judging whether the first difference value between the first probability output by the initial generator and the second probability output by the first initial discriminator, and whether the third probability output by the second initial discriminator reaches the preset target during adversarial training. This solves the problem of the difficulty in convergence and instability of generative adversarial networks.

[0021] In a second aspect, the present invention provides a device for restoring images of a dam surface, the device comprising:

[0022] The system comprises the following modules: an acquisition module for acquiring historical visible light images and historical sonar images of the dam surface; a construction module for constructing an initial generative adversarial network (GAN), which includes an initial generator, a first initial discriminator, and a second initial discriminator; a training module for iteratively training the initial generator, the first initial discriminator, and the second initial discriminator using the historical visible light images and historical sonar images to obtain a target generator, a first target discriminator, and a second target discriminator; a determination module for determining the target GAN based on the target generator, the first target discriminator, and the second target discriminator; and an acquisition and processing module for acquiring the visible light images and sonar images of the dam surface to be repaired, and inputting these images into the target GAN for processing to obtain images of the repaired dam surface.

[0023] In one alternative implementation, the training module includes:

[0024] The first input submodule is used to input the historical visible light image set and the historical sonar image set into the initial generator to obtain the fused image set; the second input submodule is used to input the fused image set and the historical visible light image set into the first initial discriminator, and input the fused image set and the historical sonar image set into the second initial discriminator; the training submodule is used to continuously perform adversarial training on the initial generator, the first initial discriminator and the second initial discriminator to obtain the target generator, the first target discriminator and the second target discriminator.

[0025] In one alternative implementation, the first input submodule includes:

[0026] The extraction unit is used to extract features from the historical visible light image set and the historical sonar image set respectively to obtain the visible light image feature set and the sonar image feature set; the fusion unit is used to fuse the visible light image feature set and the sonar image feature set to obtain the fused image feature set; the deconvolution reconstruction unit is used to perform deconvolution reconstruction on the fused image feature set to obtain the fused image set.

[0027] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the dam surface image restoration method described in the first aspect or any corresponding embodiment thereof.

[0028] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the dam surface image restoration method described in the first aspect or any corresponding embodiment thereof.

[0029] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the dam surface image restoration method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0030] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0031] Figure 1 This is a schematic flowchart of a dam surface image restoration method according to an embodiment of the present invention;

[0032] Figure 2 This is a flowchart illustrating another method for restoring images of a dam surface according to an embodiment of the present invention;

[0033] Figure 3 This is a structural block diagram of a dam surface image restoration device according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Currently, images of the dam surface collected on-site are often obscured or incomplete due to factors such as distance and environment, resulting in unclear images. This invention provides a method for restoring dam surface images by optimizing a generative adversarial network to generate high-quality images.

[0037] According to an embodiment of the present invention, a method for restoring images of a dam surface is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] This embodiment provides a method for restoring images of a dam surface, which can be used on electronic devices such as computers, mobile phones, and tablets. Figure 1 This is a flowchart of a dam surface image restoration method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0039] Step S101: Obtain a set of historical visible light images and a set of historical sonar images of the dam surface.

[0040] Specifically, the historical visible light image set and the historical sonar image set include multiple visible light images and multiple sonar images of the dam surface acquired over a historical period, respectively.

[0041] Step S102: Construct the initial generative adversarial network.

[0042] Specifically, Generative Adversarial Networks (GANs) use adversarial training to ensure that the samples generated by the generative network conform to the distribution of the real data. In a GAN, two networks are trained adversarially. One is a discriminative network, whose goal is to accurately determine whether a sample comes from real data or was generated by the generative network; the other is a generative network, whose goal is to generate samples whose source the discriminative network cannot distinguish. These two networks with opposing goals are trained alternately. When convergence occurs, if the discriminative network can no longer determine the source of a sample, it is equivalent to the generative network generating samples that conform to the distribution of the real data.

[0043] The initial generative adversarial network constructed in this embodiment may include a first generator G and two discriminators, namely a first initial discriminator Da and a second initial discriminator Db.

[0044] Step S103: Iteratively train the initial generator, the first initial discriminator, and the second initial discriminator using the historical visible light image set and the historical sonar image set to obtain the target generator, the first target discriminator, and the second target discriminator.

[0045] Specifically, historical visible light image sets and historical sonar image sets are fused into an initial generative adversarial network for game-based adversarial training until an equilibrium point is reached to obtain an optimized target generator, a first target discriminator, and a second target discriminator.

[0046] Step S104: Determine the target generation adversarial network based on the target generator, the first target discriminator, and the second target discriminator.

[0047] Specifically, a target generation adversarial network can be formed by a target generator, a first target discriminator, and a second target discriminator.

[0048] Step S105: Obtain the visible light image set and sonar image set of the dam surface to be repaired, and input the visible light image set and sonar image set into the target generative adversarial network for processing to obtain the repaired dam surface image.

[0049] Specifically, the currently acquired set of visible light images and sonar images of the dam surface to be repaired are obtained.

[0050] Furthermore, by inputting the acquired visible light image set and sonar image set into the optimized target generative adversarial network for processing, the corresponding repaired dam surface image can be output.

[0051] The dam surface image restoration method provided in this embodiment first constructs an initial generative adversarial network (GAN) comprising one generator and two discriminators. Then, by fusing visible light and sonar images into the initial GAN ​​and engaging in a game-like process to reach an equilibrium point, an optimized target GAN is obtained, solving the problems of convergence difficulty and instability in GANs. Finally, the optimized target GAN is used to restore the dam surface image, recovering the parts of the dam surface obscured by debris and generating a complete, high-quality dam surface image, thus providing support for dam maintenance.

[0052] This embodiment provides a method for restoring images of a dam surface, which can be used on electronic devices such as computers, mobile phones, and tablets. Figure 2 This is a flowchart of a dam surface image restoration method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0053] Step S201: Obtain a set of historical visible light images and a set of historical sonar images of the dam surface. For details, please refer to [link to relevant documentation]. Figure 1Step S101 of the illustrated embodiment will not be described again here.

[0054] Step S202: Construct the initial generative adversarial network. See details below. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0055] Step S203: Iteratively train the initial generator, the first initial discriminator, and the second initial discriminator using historical visible light image sets and historical sonar image sets to obtain the target generator, the first target discriminator, and the second target discriminator.

[0056] Specifically, step S203 includes:

[0057] Step S2031: Input the historical visible light image set and the historical sonar image set into the initial generator to obtain the fused image set.

[0058] Specifically, a generator is a network that can capture data distribution and generate new samples. Internally, it typically consists of multiple layers of convolutional and fully connected layers, and it employs upsampling to generate an image of appropriate size using received noise. This noise is actually random numbers that follow a certain distribution, usually a normally distributed random number. These random numbers form a vector of length n, and each such vector ultimately generates a corresponding image.

[0059] In this embodiment, by inputting the historical visible light image set and the historical sonar image set into the initial generator for processing, the historical visible light image set and the historical sonar image set can be fused to obtain the fused image set.

[0060] In some optional implementations, step S2031 above includes:

[0061] Step a1: Extract features from the historical visible light image set and the historical sonar image set respectively to obtain the visible light image feature set and the sonar image feature set.

[0062] Step a2: The visible light image feature set and the sonar image feature set are fused to obtain the fused image feature set.

[0063] Step a3: Perform deconvolution reconstruction on the fused image feature set to obtain the fused image set.

[0064] Specifically, the historical visible light image set and the historical sonar image set are input into the generator's convolutional neural network (CNN), and the features of the visible light image and the sonar image are extracted through multiple convolutional blocks to obtain the corresponding visible light image feature set and sonar image feature set.

[0065] Furthermore, the fusion of the visible light image feature set and the sonar image feature set output from the last convolutional layer can be achieved through transformation methods such as simple concatenation, linear addition, and multiplication (e.g., factor analysis, principal component analysis, PCA).

[0066] (1) Linear addition: Feature fusion in ResNet or RIR layers involves pixel-wise addition. Addition does not change the gradient. The operation is relatively simple and fast. Limitations: First, the two inputs x1 and x2 should have the same or similar meaning, or in other words, the input and output should correspond. In ResNet, one input might be 0, and that input could be one of the outputs. Direct addition also implies that the two inputs need to have the same range and equal importance.

[0067] (2) Multiplication: Generally, the weights of the feature map are obtained by using blocks, and then multiplied with the feature map to obtain a weighted effect.

[0068] (3) Concatenation: Stack the obtained feature maps together, and then perform other convolution operations. This can combine different dimensions. The purpose here is to combine features from different locations to generate new features. The main goal is to preserve the previous low-dimensional features so as not to lose too much.

[0069] Finally, the fused image feature set is reconstructed through multiple deconvolution blocks to obtain the final fused image set.

[0070] Step S2032: Input the fused image set and the historical visible light image set into the first initial discriminator, and input the fused image set and the historical sonar image set into the second initial discriminator.

[0071] Specifically, the fused image and the visible light image can be distinguished by a first initial discriminator.

[0072] Furthermore, the fused image and the sonar image can be distinguished by a second initial discriminator.

[0073] Step S2033: The target generator, the first target discriminator, and the second target discriminator are obtained by continuously performing adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator.

[0074] Specifically, by continuously performing adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator until they reach equilibrium, an optimized target generator, the first target discriminator, and the second target discriminator are generated.

[0075] In some optional implementations, step S2033 above includes:

[0076] Step b1 involves performing adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator, and obtaining the first probability output by the initial generator, the second probability output by the first initial discriminator, and the third probability output by the second initial discriminator.

[0077] Step b2: Obtain the first difference value between the first probability and the second probability, and the second difference value between the first probability and the third probability.

[0078] Step b3: Determine whether the first difference value and the second difference value have reached the preset target.

[0079] Step b4: When the first difference value and the second difference value do not reach the preset target, the adversarial training of the initial generator, the first initial discriminator and the second initial discriminator is repeatedly iterated until the first difference value and the second difference value reach the preset target, and the training stops, and the target generator, the first target discriminator and the second target discriminator are obtained.

[0080] Specifically, when the initial generator generates the fused image set f = G(a,b), the samples follow the true distribution P. data (x), the generator will generate from samples {x1,x2,…,x} m The probability distribution P of} data The probabilistic model P with learning parameter θ in (x) model (x; θ), the probability of generating sample L is defined by the following relation (1):

[0081]

[0082] Furthermore, by performing the maximum likelihood estimation shown in the following relation (2), P can be made data (x) and P model The variance between (x; θ) is minimized:

[0083]

[0084] Furthermore, the generator is a network that can capture data distribution and generate new samples. Internally, it typically consists of multiple layers of convolutional and fully connected layers, and employs upsampling to generate an image of appropriate size using received noise. This noise is essentially random numbers following a certain distribution, typically a normally distributed random number. These random numbers form a vector of length n, and each such vector ultimately generates a corresponding image.

[0085] If the input is sampled z from the latent space, it will generate a sample x = G(z), and with the help of a neural network, the probability distribution P formed by the samples generated by the generator is... G (x), the generator training objective is to make P(x), G (x) and Pdata (x) is close, and the optimization formula is shown in the following relation (3):

[0086]

[0087] Here, DIV represents the difference between the two distributions.

[0088] Furthermore, the objective function of the decision maker D is shown in the following relation (4):

[0089]

[0090] Furthermore, the objective function combining the generator and the judge is shown in the following relation (5):

[0091]

[0092] Furthermore, this example includes an initial generator and two discriminators; therefore, the corresponding objective function is shown in the following relation (6):

[0093]

[0094] Furthermore, through an initial generator G, a first initial discriminator Da, a second initial discriminator Db, and an adversarial process, P is made possible. G and two actual distributions P a and P b The differences between them both decrease simultaneously.

[0095] Furthermore, the training objective of the initial generator G is to minimize the above relation (6), and the training objective of the first initial discriminator Da and the second initial discriminator Db is to maximize the above relation (6).

[0096] Furthermore, when the minimum difference is reached simultaneously, i.e. the first difference value and the second difference value reach the equilibrium point, the optimized target generator, the first target discriminator, and the second target discriminator are obtained.

[0097] Step S204: Determine the target generation adversarial network based on the target generator, the first target discriminator, and the second target discriminator. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0098] Step S205: Obtain a set of visible light images and a set of sonar images of the dam surface to be repaired, and input these images into a target generative adversarial network for processing to obtain an image of the repaired dam surface. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0099] The dam surface image restoration method provided in this embodiment first extracts features from historical visible light image sets and historical sonar image sets respectively. Second, the extracted visible light image feature sets and sonar image feature sets are fused. Finally, the fused image feature set is deconvolutionally reconstructed to obtain the final fused image set. Further, the fused image set and the historical visible light image set are input into a first initial discriminator to discriminate the visible light images, and then input into a second initial discriminator to discriminate the sonar images. Finally, by continuously performing adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator, and by judging whether the first difference between the first probability output by the initial generator and the second probability output by the first initial discriminator, and the third probability output by the second initial discriminator (comparing the first probability and the third probability output by the second initial discriminator) reach a preset target during the adversarial training process, it can be further determined whether to continue adversarial training. Finally, the optimized target generative adversarial network was used to repair the dam surface image, restoring the parts of the dam surface that were obscured by debris and generating a complete, high-quality image of the dam surface, which provides support for dam maintenance.

[0100] This embodiment also provides a dam surface image restoration device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0101] This embodiment provides a device for restoring images of a dam surface, such as... Figure 3 As shown, it includes:

[0102] The acquisition module 301 is used to acquire historical visible light image sets and historical sonar image sets of the dam surface.

[0103] The construction module 302 is used to construct an initial generative adversarial network, which includes an initial generator, a first initial discriminator, and a second initial discriminator.

[0104] Training module 303 is used to iteratively train the initial generator, the first initial discriminator and the second initial discriminator using historical visible light image set and historical sonar image set to obtain target generator, first target discriminator and second target discriminator.

[0105] The determination module 304 is used to determine the target generation adversarial network based on the target generator, the first target discriminator, and the second target discriminator.

[0106] The acquisition and processing module 305 is used to acquire a set of visible light images and a set of sonar images of the dam surface to be repaired, and input the set of visible light images and the set of sonar images into the target generative adversarial network for processing to obtain the image of the dam surface after repair.

[0107] In some alternative implementations, training module 303 includes:

[0108] The first input submodule is used to input the historical visible light image set and the historical sonar image set into the initial generator to obtain the fused image set.

[0109] The second input submodule is used to input the fused image set and the historical visible light image set into the first initial discriminator, and to input the fused image set and the historical sonar image set into the second initial discriminator.

[0110] The training submodule is used to obtain the target generator, the first target discriminator, and the second target discriminator by continuously performing adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator.

[0111] In some alternative implementations, the first input submodule includes:

[0112] The extraction unit is used to extract features from the historical visible light image set and the historical sonar image set, respectively, to obtain the visible light image feature set and the sonar image feature set.

[0113] The fusion unit is used to fuse the visible light image feature set and the sonar image feature set to obtain a fused image feature set.

[0114] The deconvolution reconstruction unit is used to perform deconvolution reconstruction on the feature set of the fused image to obtain the fused image set.

[0115] In some alternative implementations, the training submodule includes:

[0116] The training and acquisition unit is used to perform adversarial training on the initial generator, the first initial discriminator, and the second initial discriminator, and to acquire the first probability output by the initial generator, the second probability output by the first initial discriminator, and the third probability output by the second initial discriminator.

[0117] The acquisition unit is used to acquire a first difference value between the first probability and the second probability, and a second difference value between the first probability and the third probability;

[0118] The judgment unit is used to determine whether the first difference value and the second difference value have reached the preset target.

[0119] The iterative unit is used to iterate the adversarial training of the initial generator, the first initial discriminator, and the second initial discriminator repeatedly until the first and second difference values ​​reach the preset target, and then stop training to obtain the target generator, the first target discriminator, and the second target discriminator.

[0120] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0121] In this embodiment, the dam surface image restoration device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0122] This invention also provides a computer device having the above-described features. Figure 3 The image restoration device for the dam surface shown.

[0123] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0124] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0125] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0126] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0127] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0128] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0129] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0130] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0131] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method of inpainting a dam surface image, characterized by, The method comprises: acquiring a historical visible light image set and a historical sonar image set of a dam surface; constructing an initial generative adversarial network comprising an initial generator, a first initial discriminator and a second initial discriminator; iteratively training the initial generator, the first initial discriminator and the second initial discriminator using the historical visible light image set and the historical sonar image set and obtaining a target generator, a first target discriminator and a second target discriminator; determining a target generative adversarial network according to the target generator, the first target discriminator and the second target discriminator; acquiring a visible light image set and a sonar image set of a dam surface to be repaired and inputting the visible light image set and the sonar image set into the target generative adversarial network for processing to obtain a repaired dam surface image; wherein iteratively training the initial generator, the first initial discriminator and the second initial discriminator using the historical visible light image set and the historical sonar image set and obtaining a target generator, a first target discriminator and a second target discriminator comprises: inputting the historical visible light image set and the historical sonar image set into the initial generator to obtain a fused image set; inputting the fused image set and the historical visible light image set into the first initial discriminator and inputting the fused image set and the historical sonar image set into the second initial discriminator; continuously performing adversarial training on the initial generator, the first initial discriminator and the second initial discriminator to obtain the target generator, the first target discriminator and the second target discriminator.

2. The method of claim 1, wherein, inputting the historical visible light image set and the historical sonar image set into the initial generator to obtain a fused image set comprises: respectively performing feature extraction on the historical visible light image set and the historical sonar image set to obtain a visible light image feature set and a sonar image feature set; fusing the visible light image feature set and the sonar image feature set to obtain a fused image feature set; performing deconvolution reconstruction on the fused image feature set to obtain the fused image set.

3. The method of claim 1, wherein, continuously performing adversarial training on the initial generator, the first initial discriminator and the second initial discriminator to obtain the target generator, the first target discriminator and the second target discriminator comprises: performing adversarial training on the initial generator, the first initial discriminator and the second initial discriminator and acquiring a first probability output by the initial generator, a second probability output by the first initial discriminator and a third probability output by the second initial discriminator; acquiring a first difference value of the first probability and the second probability and a second difference value of the first probability and the third probability; determining whether the first difference value and the second difference value reach a preset target; and continuously performing adversarial training on the initial generator, the first initial discriminator and the second initial discriminator until the first difference value and the second difference value reach the preset target. When the first difference value and the second difference value do not reach the preset target, the adversarial training of the initial generator, the first initial discriminator and the second initial discriminator is iterated repeatedly until the training is stopped when the first difference value and the second difference value reach the preset target, and the target generator, the first target discriminator and the second target discriminator are obtained.

4. A dam surface image inpainting apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire a historical visible light image set and a historical sonar image set of a dam surface; A construction module is configured to construct an initial generative adversarial network, wherein the initial generative adversarial network comprises an initial generator, a first initial discriminator and a second initial discriminator; A training module is configured to iteratively train the initial generator, the first initial discriminator and the second initial discriminator by using the historical visible light image set and the historical sonar image set, and obtain a target generator, a first target discriminator and a second target discriminator; A determination module is configured to determine a target generative adversarial network according to the target generator, the first target discriminator and the second target discriminator; An acquisition and processing module is configured to acquire a visible light image set and a sonar image set of a dam surface to be repaired, and input the visible light image set and the sonar image set into the target generative adversarial network for processing to obtain a repaired dam surface image; The training module comprises: A first input sub-module is configured to input the historical visible light image set and the historical sonar image set into the initial generator to obtain a fused image set; A second input sub-module is configured to input the fused image set and the historical visible light image set into the first initial discriminator, and input the fused image set and the historical sonar image set into the second initial discriminator; A training sub-module is configured to obtain the target generator, the first target discriminator and the second target discriminator by continuously performing adversarial training on the initial generator, the first initial discriminator and the second initial discriminator.

5. The apparatus of claim 4, wherein, The first input sub-module comprises: An extraction unit is configured to respectively perform feature extraction on the historical visible light image set and the historical sonar image set to obtain a visible light image feature set and a sonar image feature set; A fusion unit is configured to fuse the visible light image feature set and the sonar image feature set to obtain a fused image feature set; A deconvolution reconstruction unit is configured to perform deconvolution reconstruction on the fused image feature set to obtain the fused image set.

6. A computer device, comprising: It comprises: A memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the dam surface image repair method in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the dam surface image repair method in any one of claims 1 to 3.

8. A computer program product, characterised in that, It comprises computer instructions, and the computer instructions are used to make a computer execute the dam surface image repair method in any one of claims 1 to 3.

Images