Dam surface image restoration method, device and equipment and storage medium
By introducing a self-attention mechanism module into the adversarial generation network, the training process is optimized, and the problem of difficulty in convergence of GAN is solved, high-quality repair of dam surface images is achieved, and dam maintenance is provided.
Patent Information
- Application Number
- CN202410940037.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the GAN of the adversarial neural network model is difficult to converge and unstable, resulting in poor repair of the image on the dam surface.
The self-attention mechanism module is added to the initial adversarial generation network, and the optimized target adversarial generation network is generated through iterative adversarial training to process and repair dam surface images.
The problem of difficulty convergence and instability of the generative adversarial network was solved, and a complete and high-quality dam surface image was generated to support the maintenance and repair of the dam.
Smart Images

Figure CN118918043A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a method, device, equipment and storage medium for repairing a dam surface image. Background Art
[0002] At present, due to environmental factors, underwater image collection on the dam surface is blocked by debris such as water plants, and the collected images are incomplete and unclear. The adversarial neural network model GAN can be used to process the image and restore it.
[0003] GAN consists of two important parts: Generator (G) and Discriminator (D).
[0004] Generator: Generates data through a machine, with the goal of "cheating" the discriminator as much as possible. The generated data is recorded as G(z);
[0005] Discriminator: Determines whether the data is real data or data generated by the "generator". The purpose is to find out the "fake data" created by the "generator" as much as possible. Its input parameter is x, x represents the data, and the output D(x) represents the probability that x is real data. If it is 1, it means that it is 100% real data, and if the output is 0, it means that it cannot be real data.
[0006] G and D form a dynamic confrontation (or game process). As the training (confrontation) progresses, the data generated by G becomes closer and closer to the real data, and the level of data identification by D becomes higher and higher. In an ideal state, G can generate data that is "indistinguishable from the real"; but for D, it is difficult to determine whether the data generated by the generator is real, so D(G(z)) = 0.5. After the training is completed, a generative model G can be obtained, which can be used to generate data that is indistinguishable from the real.
[0007] However, in the existing technology, the adversarial neural network model GAN is difficult to converge and unstable. The generator and the discriminator need to be well synchronized, but in actual training, it is easy for D to converge and G to diverge. Summary of the invention
[0008] In view of this, the present invention provides a dam surface image restoration method, device, equipment and storage medium to solve the problem of difficulty in convergence and instability of the adversarial neural network model GAN in the prior art.
[0009] In a first aspect, the present invention provides a method for repairing a dam surface image, the method comprising:
[0010] An initial adversarial generation network, a random image feature vector set and a historical image data set of the dam surface are obtained, wherein the initial adversarial generation network includes a generator and a discriminator; a self-attention mechanism module is added to the initial adversarial generation network to obtain a target adversarial generation network; the random image feature vector set is processed by the self-attention mechanism module to obtain a target image feature vector set; the target image feature vector set and the historical image data set are used to perform iterative adversarial training on the generator and the discriminator and generate a target adversarial generation network; a measured image data set of the dam surface to be repaired is obtained, and the measured image data set is processed by the target adversarial generation network to obtain an image of the repaired dam surface.
[0011] The dam surface image restoration method provided by the present invention obtains a second adversarial generation network by adding a self-attention mechanism module to the first generation adversarial network. Further, the output of the self-attention mechanism module is used as the input of the generator and iterative adversarial training is performed with the discriminator until an optimized target adversarial generation network is generated, thereby solving the problem that the generation adversarial network is difficult to converge and unstable. Finally, the dam surface image is restored by the optimized target generation adversarial network, the part of the dam surface blocked by debris is restored, and a complete high-quality dam surface image is generated, which provides support for the maintenance of the dam.
[0012] In an optional implementation, the random image feature vector set is processed by a self-attention mechanism module to obtain a target image feature vector set, including:
[0013] Based on the random image feature vector set, multiple attention weight values are obtained through calculation by the self-attention mechanism module; based on the multiple attention weight values, the target image feature vector set is determined in the random image feature vector set.
[0014] The dam surface image restoration method provided by the present invention can calculate multiple attention weight values through the self-attention mechanism module, and then combine the multiple attention weight values to screen out the corresponding target image feature vector set in the random image feature vector set, thereby providing support for the subsequent optimization training of the target generative adversarial network.
[0015] In an optional implementation, based on a random image feature vector set, a self-attention mechanism module calculates and obtains multiple attention weight values, including:
[0016] Based on a set of random image feature vectors, multiple target Query values, multiple target Key values and multiple target Value values are calculated respectively; multiple similarity values are calculated according to the multiple target Query values and the multiple target Key values; the multiple similarity values are normalized to obtain weight coefficient values; and multiple attention weight values are calculated according to the multiple target Value values and the weight coefficient values.
[0017] The dam surface image restoration method provided by the present invention can calculate multiple attention weight values by calculating the Query value, the Key value and the Value value, thereby providing support for the subsequent screening of the target image feature vector set.
[0018] In an optional implementation, based on the random image feature vector set, multiple target Query values, multiple target Key values, and multiple target Value values are calculated respectively, including:
[0019] According to a set of random image feature vectors, a preset first weight matrix and a preset second weight matrix, multiple target Query values, multiple target Key values and multiple initial Value values are calculated; according to the multiple target Query values and the multiple target Key values, attention score values are calculated; according to the attention score values, multiple target Value values are determined from the multiple initial Value values.
[0020] In an optional implementation, the target image feature vector set and the historical image data set are used to perform iterative adversarial training on the generator and the discriminator and generate a target adversarial generation network, including:
[0021] The target image feature vector set is input into the generator to obtain the target image dataset; the target image dataset and the historical image dataset are input into the discriminator and adversarially trained with the generator until the objective function is met, thereby generating a target adversarial generation network.
[0022] The dam surface image restoration method provided by the present invention inputs the target image data set output by the generator and the real historical image data set into the discriminator respectively and performs adversarial training with the generator until the objective function is met, stops the training and generates an optimized target adversarial generative network, solves the problem that the generative adversarial network is difficult to converge and unstable, and provides support for the subsequent restoration of the dam surface image.
[0023] In a second aspect, the present invention provides a dam surface image restoration device, the device comprising:
[0024] An acquisition module is used to acquire an initial adversarial generation network, a random image feature vector set and a historical image data set of the dam surface, wherein the initial adversarial generation network includes a generator and a discriminator; an adding module is used to add a self-attention mechanism module to the initial adversarial generation network to obtain a target adversarial generation network; a processing module is used to process the random image feature vector set through the self-attention mechanism module to obtain a target image feature vector set; a training module is used to perform iterative adversarial training on the generator and the discriminator using the target image feature vector set and the historical image data set and generate a target adversarial generation network; an acquisition and training module is used to acquire a measured image data set of the dam surface to be repaired, and process the measured image data set through the target adversarial generation network to obtain an image of the repaired dam surface.
[0025] In an optional embodiment, the processing module includes:
[0026] The calculation submodule is used to obtain multiple attention weight values based on the random image feature vector set after calculation by the self-attention mechanism module; the determination submodule is used to determine the target image feature vector set in the random image feature vector set based on the multiple attention weight values.
[0027] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the dam surface image repair method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the dam surface image restoration method of the first aspect or any corresponding embodiment thereof.
[0029] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the dam surface image restoration method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 is a schematic flow chart of a dam surface image restoration method according to an embodiment of the present invention;
[0032] Figure 2 is a flow chart of another dam surface image restoration method according to an embodiment of the present invention;
[0033] Figure 3 is a schematic diagram of the attention score value calculation process according to an embodiment of the present invention;
[0034] Figure 4 is b according to an embodiment of the present invention 1 Generate schematic diagrams;
[0035] Figure 5 is a schematic flow chart of another dam surface image restoration method according to an embodiment of the present invention;
[0036] Figure 6 is a structural block diagram of a dam surface image restoration device according to an embodiment of the present invention;
[0037] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0039] At present, due to factors such as distance and environment, the dam surface images collected on site are blocked or cannot be fully photographed, so it is impossible to obtain a clear image. An embodiment of the present invention provides a dam surface image restoration method, which optimizes the generative adversarial network to achieve the effect of generating high-quality pictures.
[0040] According to an embodiment of the present invention, an embodiment of a dam surface image restoration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0041] In this embodiment, a dam surface image restoration method is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 FIG. 4 is a flow chart of a dam surface image restoration method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0042] Step S101, obtaining a first generative adversarial network, a random image feature vector set, and a historical image dataset of the dam surface.
[0043] The first generative adversarial network may include a generator and a discriminator.
[0044] Specifically, the generative adversarial network (GAN) uses adversarial training to make the samples generated by the generative network conform to the real data distribution. In the generative adversarial network, there are two networks for adversarial training. One is the discriminator network, whose goal is to judge as accurately as possible whether a sample comes from real data or is generated by the generative network; the other is the generator network, whose goal is to generate samples whose sources cannot be distinguished by the discriminator network. These two networks with opposite goals are constantly trained alternately. When they finally converge, if the discriminator network can no longer determine the source of a sample, then it is equivalent to the generative network being able to generate samples that conform to the real data distribution.
[0045] Furthermore, the random image feature vector set may include a plurality of feature vectors extracted from the acquired dam surface test set images.
[0046] Furthermore, the historical image data set may include a plurality of real images of the dam surface acquired within a historical period of time.
[0047] Step S102, adding a self-attention mechanism module to the first generative adversarial network to obtain a second generative adversarial network.
[0048] Among them, the self-attention mechanism module can assign different weights according to different parts of the input data. Its application in the generator can make the generator pay more attention to the important parts when generating data, thereby generating more realistic and high-quality data. It has the following advantages:
[0049] (1) Focus on key information: It can selectively focus on the important parts of the input data and ignore irrelevant or minor information, thereby improving the model's ability to extract and process key features.
[0050] (2) Enhanced model expression capability: By dynamically assigning weights, the model can adaptively learn the importance of different positions or features, enriching the model’s expression and learning methods.
[0051] (3) Improve model performance: It helps to improve the accuracy, efficiency and generalization ability of the model in various tasks.
[0052] (4) Better processing of long sequence data: When processing long sequence data such as long texts and long videos, it can effectively capture global dependencies and avoid information loss.
[0053] (5) Enhanced interpretability: It provides certain clues for understanding the decision-making process of the model, making the model's behavior more interpretable.
[0054] Specifically, in this embodiment, adding a self-attention mechanism module to the generative adversarial network can improve the performance of the generator, enabling it to pay more attention to important areas or features when generating data, thereby improving the quality of the generated data.
[0055] Step S103, processing the random image feature vector set through the self-attention mechanism module to obtain a first image set.
[0056] Specifically, by inputting a random image feature vector set into the self-attention mechanism module for processing, the corresponding first image set can be output.
[0057] In one example, taking SENet (Squeeze-and-Excitation Network) in the channel self-attention mechanism as an example, a specific implementation method for generating the first image set is given:
[0058] (1) Perform global average pooling on the random image feature vector set, compressing the dimension of the feature map from (c, h, w) to (c, 1, 1). In other words, for each channel c, a corresponding number is obtained. The purpose of this step is to compress the information of each channel into a value with a global receptive field.
[0059] (2) Then, two full connection operations are performed. The first full connection reduces the number of channels (for example, to c / r, where r is a scaling ratio), which can reduce the amount of calculation and capture the nonlinear relationship between channels. The second full connection restores the number of channels to the original c.
[0060] (3) Apply the Sigmoid function to the output of the second full connection, limit the value between 0 and 1, and obtain the weight of each channel. These weights represent the importance of each channel.
[0061] (4) Finally, the original random image feature vector set is multiplied by the obtained channel weight to achieve recalibration of features of different channels. In this way, the first image set processed by the self-attention mechanism is obtained.
[0062] In this way, Senet allows the model to pay more attention to important channel features and suppress less important channel features, thereby improving the model's image representation ability and performance.
[0063] Step S104, using the first image set and the historical image data set to perform iterative adversarial training on the generator and discriminator of the second generative adversarial network and generate a target generative adversarial network.
[0064] Specifically, the first image set and the historical image data set are used as training data. The generator is continuously allowed to generate new images, and the discriminator judges the authenticity of these generated images. The two are confronted and optimized with each other. After multiple iterations, a target adversarial generative network that can generate images that meet the requirements is finally obtained.
[0065] In iterative adversarial training, the generator strives to generate realistic images, making it difficult for the discriminator to distinguish between real images and generated images; the discriminator strives to improve its own discrimination ability and accurately distinguish between real images and images generated by the generator. As the training progresses, the performance of the generator and the discriminator continues to improve, eventually reaching a state of equilibrium and generating the corresponding optimized target adversarial generative network.
[0066] Step S105, obtaining a measured image dataset of the surface of the dam to be repaired, and processing the measured image dataset through a target adversarial generation network to obtain an image of the repaired dam surface.
[0067] Specifically, a measured image dataset of the surface of the dam to be repaired can be obtained through a specific measurement method, which can include image information of the actual conditions of various damages and defects on the dam surface.
[0068] Furthermore, the obtained measured image dataset is input into a trained target adversarial generative network, which processes and transforms the input measured image dataset based on its learned patterns and features.
[0069] Furthermore, after the calculation and generation process of the target adversarial generative network, the repaired dam surface image is finally output. These generated images can theoretically present the ideal state of the dam surface after repair, and can provide reference and guidance for subsequent dam repair work, such as helping to formulate repair plans and evaluate repair effects.
[0070] The dam surface image restoration method provided in this embodiment obtains a second adversarial generation network by adding a self-attention mechanism module to the first generation adversarial network. Further, the output of the self-attention mechanism module is used as the input of the generator and iterative adversarial training is performed with the discriminator until an optimized target adversarial generation network is generated, thereby solving the problem that the generation adversarial network is difficult to converge and unstable. Finally, the dam surface image is restored by the optimized target generation adversarial network, and the part of the dam surface blocked by debris is restored, and a complete high-quality dam surface image is generated, which provides support for the maintenance of the dam.
[0071] In this embodiment, a dam surface image restoration method is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 2 FIG. 4 is a flow chart of a dam surface image restoration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0072] Step S201, obtain a first generative adversarial network, a random image feature vector set and a historical image dataset of the dam surface. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0073] Step S202: Add a self-attention mechanism module to the first generative adversarial network to obtain a second generative adversarial network. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0074] Step S203: Process the random image feature vector set through a self-attention mechanism module to obtain a first image set.
[0075] Specifically, the above step S203 includes:
[0076] Step S2031, based on the random image feature vector set, a plurality of attention weight values are obtained through calculation by the self-attention mechanism module.
[0077] Specifically, the self-attention mechanism module can assign different weights according to the importance or relevance of feature vectors in the random image feature vector set, and can evaluate and calculate various properties of feature vectors (such as numerical size, distribution, relationship with other vectors, etc.).
[0078] Among them, the larger the attention weight value, the more important or more concerned the corresponding feature vector is in the subsequent processing or task.
[0079] In some optional implementations, the above step S2031 includes:
[0080] Step a1, based on the random image feature vector set, calculate multiple target Query values, multiple target Key values and multiple target Value values respectively.
[0081] Specifically, the self-attention mechanism module can realize the attention weight allocation of Value and generate the final output by converging the attention of Query and Key (given a Query, calculate the correlation between Query and Key, and then find the most appropriate Value based on the correlation between Query and Key).
[0082] In some optional implementations, the above step a1 includes:
[0083] Step a11, calculating multiple target Query values, multiple target Key values and multiple initial Value values according to the random image feature vector set, the preset first weight matrix and the preset second weight matrix.
[0084] Step a12, calculating the attention score value according to multiple target Query values and multiple target Key values.
[0085] Step a13, determining multiple target Value values from multiple initial Value values according to the attention score value.
[0086] Specifically, the feature vector a is concentrated in the random image feature vector 1 and a 2 Take the above calculation process as an example to explain.
[0087] like Figure 3 As shown, first, the input feature vector a 1 With the preset first weight matrix W q Multiplying them together gives a vector q, which is the Query value.
[0088] Secondly, the input feature vector a 2 and the preset second weight matrix W k Multiply them together to get a vector k, which is the Key value.
[0089] Then, we do a dot product of q and k to get α (attention score), which is the attention score value. α represents the degree of correlation between the two vectors.
[0090] Further, if Figure 4 As shown in the figure, the calculation method of Value is the same as that of Query value and Key value. The input feature vector a is multiplied by the weight matrix W to obtain v (i.e. Value value), which is then multiplied by the corresponding α′. Each v is multiplied by α′ and then summed to obtain the output b. 1 .
[0091] Furthermore, if a 1 and a 2 The correlation is relatively high, α′ 1,2 is relatively large, then the output b 1 It may be closer to v 2 , that is, α (attention score) determines the weight of the vector in the result.
[0092] Therefore, multiple target Value values can be determined from multiple initial Value values through the attention score value α.
[0093] Furthermore, b can be expressed using matrix operations 1 The generation of is shown in the following relation (1):
[0094]
[0095] Step a2, calculating multiple similarity values according to multiple target Query values and multiple target Key values.
[0096] Specifically, the similarity or correlation sim between the two can be calculated based on multiple target Query values and multiple target Key values, as shown in the following equation (2):
[0097] sim(Query,Key i )=Query·Key i (2)
[0098] Step a3, normalizing multiple similarity values to obtain weight coefficient values.
[0099] Specifically, we can use softmax to calculate the sim(Query,Key i ) is normalized and the weight coefficient value A is obtained i , as shown in the following relation (3):
[0100] A i =Softmax(sim i )(3)
[0101] Step a4, calculate multiple attention weight values according to multiple target Value values and weight coefficient values.
[0102] Specifically, the attention weight value Attention is calculated using the following relationship (4):
[0103]
[0104] Step S2032: determining a target image feature vector set from a random image feature vector set based on multiple attention weight values.
[0105] Specifically, according to the obtained Attention, a more important target image feature vector set can be screened out from the random image feature vector set.
[0106] Step S204, using the first image set and the historical image data set to iteratively train the generator and discriminator of the second adversarial generation network and generate a target adversarial generation network. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.
[0107] Step S205, obtain a measured image dataset of the surface of the dam to be repaired, and process the measured image dataset through a target adversarial generative network to obtain a repaired dam surface image. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.
[0108] The dam surface image restoration method provided in this embodiment first obtains a second adversarial generative network by adding a self-attention mechanism module to the first generative adversarial network. Secondly, multiple attention weight values can be calculated by calculating the Query value, the Key value, and the Value value, and then the corresponding target image feature vector set can be screened out from the random image feature vector set in combination with the multiple attention weight values. Finally, the target image feature vector set output by the self-attention mechanism module is used as the input of the generator and iterative adversarial training is performed with the discriminator until an optimized target adversarial generative network is generated, which solves the problem that the generative adversarial network is difficult to converge and unstable. Finally, the dam surface image is restored by the optimized target generative adversarial network, the part of the dam surface blocked by debris is restored, and a complete and high-quality dam surface image is generated, which provides support for the maintenance of the dam.
[0109] In this embodiment, a dam surface image restoration method is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 5 FIG. 4 is a flow chart of a dam surface image restoration method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:
[0110] Step S501, obtain a first generative adversarial network, a random image feature vector set and a historical image dataset of the dam surface. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0111] Step S502: Add a self-attention mechanism module to the first generative adversarial network to obtain a second generative adversarial network. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0112] Step S503: Process the random image feature vector set through the self-attention mechanism module to obtain the first image set. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.
[0113] Step S504, using the first image set and the historical image data set to perform iterative adversarial training on the generator and discriminator of the second generative adversarial network and generate a target generative adversarial network.
[0114] Specifically, the above step S504 includes:
[0115] Step S5041, input the target image feature vector set into the generator to obtain the target image data set.
[0116] Specifically, the target image feature vector set is input into the generator, and the corresponding image is generated through the neural network structure and learned parameters of the generator.
[0117] Among them, the generator is a network that can capture data distribution and generate new samples. Its internal structure is generally a network composed of multiple layers of convolution, fully connected layers, etc. Each input vector will eventually generate a corresponding image for the discriminator to identify.
[0118] Step S5042, input the target image data set and the historical image data set into the discriminator, and perform adversarial training with the generator until the objective function is met, and then generate a target adversarial generation network.
[0119] Specifically, the generator G maps data from an input space to a generated space (i.e., generates output data through input data under the action of a function), and the formula is x=G(z). The input z satisfies a simple form of random distribution (such as Gaussian distribution or uniform distribution, etc.). In order to make the generated data distribution as close as possible to the real data distribution, the generating function G will be in the form of a neural network, through which completely different distribution types can be simulated.
[0120] Furthermore, the objective function is expressed as follows:
[0121]
[0122] Where x represents a real image, i.e., an image in the historical image dataset; z represents the noise input to the generator G; G(z) represents the image generated by the generator G, i.e., an image in the target image dataset; D(x) represents the probability that the discriminator D judges whether the real image x is real; D(G(z)) represents the probability that the discriminator D judges whether the image G(z) generated by the generator G is real.
[0123] The purpose of the generator G is to generate images that are as close to reality as possible.
[0124] The purpose of the discriminator D is: the stronger the ability of D is, the larger D(x) should be, and the smaller D(G(z)) should be. At this time, V(D,G) will become larger. Therefore, the above relationship (5) is to find the maximum value (max_D) for the discriminator D.
[0125] Furthermore, in order to speed up training in the early stage, the training of the generator G can replace the log(1-D(G(z))) with the -logD(G(z)) loss function.
[0126] Specifically, the above equation (5) consists of two terms. max() is the first step, which means that the generator G is fixed and the discriminator is allowed to distinguish between real samples and fake samples as much as possible. That is, it is hoped that the discriminator can distinguish between real samples and generated samples without changing the generator.
[0127] Furthermore, min() is the second step. The discriminator D is fixed. By adjusting the generator, we hope that the discriminator will make mistakes and try not to let it distinguish.
[0128] Furthermore, when optimizing the maximum minimization objective function shown in the above relation (5), the generator G and the discriminator D are iterated alternately. For a period of time, the parameters in the generator G are fixed to optimize the discriminator D. For another period of time, the parameters in the discriminator D are fixed to optimize the parameters in the generator G, thereby training a target adversarial generative network that meets the above objective function:
[0129] (1) Fix the generator G and train the discriminator D: Select real images from the target image dataset and the historical image dataset, and let the generator G generate a batch of fake images, which are input into the discriminator D. Calculate the loss of the discriminator D, and update the parameters of the discriminator D through back propagation to improve its discrimination ability.
[0130] (2) Fix the discriminator D and train the generator G: Let the generator G generate a batch of fake images again and input them into the discriminator D. Calculate the loss of the generator G, and then update the parameters of the generator G through back propagation so that it can generate more realistic images that can deceive the discriminator D.
[0131] (3) Repeat the alternating training steps: Repeat the above process of fixing the generator to train the discriminator and fixing the discriminator to train the generator. The two networks are gradually optimized in the confrontation and the target adversarial generative network that satisfies the above objective function is obtained.
[0132] Step S505, obtain a measured image dataset of the surface of the dam to be repaired, and process the measured image dataset through a target adversarial generation network to obtain a repaired dam surface image. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.
[0133] The dam surface image restoration method provided in this embodiment obtains a second adversarial generative network by adding a self-attention mechanism module to the first generative adversarial network. Further, the output of the self-attention mechanism module is used as the input of the generator, and the target image data set output by the generator and the real historical image data set are respectively input into the discriminator and adversarially trained with the generator until the objective function is met, the training is stopped and an optimized target adversarial generative network is generated, thereby solving the problem that the generative adversarial network is difficult to converge and unstable. Finally, the dam surface image is restored by the optimized target generative adversarial network, the part of the dam surface blocked by debris is restored, and a complete and high-quality dam surface image is generated, which provides support for the maintenance of the dam.
[0134] In this embodiment, a dam surface image restoration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0135] This embodiment provides a dam surface image restoration device, such as Figure 6 As shown, including:
[0136] The acquisition module 601 is used to acquire an initial adversarial generation network, a random image feature vector set and a historical image data set of the dam surface, wherein the initial adversarial generation network includes a generator and a discriminator.
[0137] Adding module 602 is used to add a self-attention mechanism module to the initial adversarial generation network to obtain a target adversarial generation network.
[0138] The processing module 603 is used to process the random image feature vector set through the self-attention mechanism module to obtain the target image feature vector set.
[0139] The training module 604 is used to perform iterative adversarial training on the generator and the discriminator using the target image feature vector set and the historical image data set and generate a target adversarial generation network.
[0140] The acquisition and training module 605 is used to acquire a measured image dataset of the surface of the dam to be repaired, and process the measured image dataset through a target adversarial generation network to obtain an image of the repaired dam surface.
[0141] In some optional implementations, the processing module 603 includes:
[0142] The calculation submodule is used to obtain multiple attention weight values based on a random image feature vector set through calculation by the self-attention mechanism module.
[0143] The determination submodule is used to determine a target image feature vector set from a random image feature vector set based on a plurality of attention weight values.
[0144] In some optional implementations, the calculation submodule includes:
[0145] The first calculation unit is used to calculate multiple target Query values, multiple target Key values and multiple target Value values respectively based on the random image feature vector set.
[0146] The second calculation unit is used to calculate multiple similarity values according to multiple target Query values and multiple target Key values.
[0147] The processing unit is used to perform normalization processing on multiple similarity values to obtain a weight coefficient value.
[0148] The third calculation unit is used to calculate multiple attention weight values according to multiple target Value values and weight coefficient values.
[0149] In some optional implementations, the first computing unit includes:
[0150] The first calculation subunit is used to calculate multiple target Query values, multiple target Key values and multiple initial Value values according to the random image feature vector set, the preset first weight matrix and the preset second weight matrix.
[0151] The second calculation subunit is used to calculate the attention score value according to multiple target Query values and multiple target Key values.
[0152] A determination subunit is used to determine multiple target Value values from multiple initial Value values according to the attention score value.
[0153] In some optional implementations, the training module 604 includes:
[0154] The input submodule is used to input the target image feature vector set into the generator to obtain the target image dataset.
[0155] The training submodule is used to input the target image dataset and the historical image dataset into the discriminator and perform adversarial training with the generator until the objective function is met, thereby generating a target adversarial generation network.
[0156] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0157] The dam surface image restoration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0158] The embodiment of the present invention also provides a computer device having the above Figure 6 The dam surface image restoration device is shown.
[0159] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.
[0160] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0161] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0162] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0163] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0164] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0165] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0166] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0167] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A dam surface image restoration method, characterized in that: The method comprises: Obtaining an initial adversarial generation network, a random image feature vector set, and a historical image dataset of the dam surface, wherein the initial adversarial generation network includes a generator and a discriminator; Adding a self-attention mechanism module to the initial generative adversarial network to obtain a target generative adversarial network; Processing the random image feature vector set through a self-attention mechanism module to obtain a target image feature vector set; Using the target image feature vector set and the historical image data set to iteratively train the generator and the discriminator and generate a target adversarial generation network; A measured image dataset of the surface of the dam to be repaired is obtained, and the measured image dataset is processed by the target adversarial generation network to obtain a repaired dam surface image.
2. The method according to claim 1, characterized in that The random image feature vector set is processed by the self-attention mechanism module to obtain a target image feature vector set, including: Based on the random image feature vector set, a plurality of attention weight values are obtained through calculation by the self-attention mechanism module; Based on the multiple attention weight values, the target image feature vector set is determined in the random image feature vector set.
3. The method according to claim 2, characterized in that Based on the random image feature vector set, multiple attention weight values are obtained through calculation by the self-attention mechanism module, including: Based on the random image feature vector set, a plurality of target Query values, a plurality of target Key values and a plurality of target Value values are calculated respectively; Calculating multiple similarity values according to the multiple target Query values and the multiple target Key values; Normalizing the multiple similarity values to obtain a weight coefficient value; The multiple attention weight values are calculated based on the multiple target Value values and the weight coefficient values.
4. The method according to claim 3, characterized in that Based on the random image feature vector set, a plurality of target Query values, a plurality of target Key values, and a plurality of target Value values are calculated respectively, including: Calculating the plurality of target Query values, the plurality of target Key values, and the plurality of initial Value values according to the random image feature vector set, the preset first weight matrix, and the preset second weight matrix; Calculate the attention score value according to the multiple target Query values and the multiple target Key values; The multiple target Value values are determined from the multiple initial Value values according to the attention score value.
5. The method according to claim 1, characterized in that The target image feature vector set and the historical image data set are used to perform iterative adversarial training on the generator and the discriminator and generate a target adversarial generation network, including: Inputting the target image feature vector set into the generator to obtain a target image data set; The target image data set and the historical image data set are input into the discriminator, and adversarial training is performed with the generator until the objective function is satisfied, thereby generating the target adversarial generation network.
6. A dam surface image restoration device, characterized in that: The device comprises: An acquisition module is used to acquire an initial adversarial generation network, a random image feature vector set, and a historical image data set of the dam surface, wherein the initial adversarial generation network includes a generator and a discriminator; An adding module is used to add a self-attention mechanism module to the initial adversarial generation network to obtain a target adversarial generation network; A processing module, used for processing the random image feature vector set through a self-attention mechanism module to obtain a target image feature vector set; A training module, used for performing iterative adversarial training on the generator and the discriminator using the target image feature vector set and the historical image data set and generating a target adversarial generation network; The acquisition and training module is used to acquire a measured image dataset of the surface of the dam to be repaired, and process the measured image dataset through the target adversarial generation network to obtain the repaired dam surface image.
7. The device according to claim 6, characterized in that The processing module comprises: A calculation submodule, used for obtaining a plurality of attention weight values based on the random image feature vector set through calculation by the self-attention mechanism module; A determination submodule is used to determine the target image feature vector set in the random image feature vector set based on the multiple attention weight values.
8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the dam surface image restoration method according to any one of claims 1 to 5 by executing the computer instructions.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the dam surface image repair method according to any one of claims 1 to 5.
10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the dam surface image repair method according to any one of claim 5.
Citation Information
Patent Citations
Image restoration method based on adversarial generative network
CN111612708A
