Road disease prediction method and device based on image generative adversarial network

Abstract

The invention relates to the technical field of data processing and artificial intelligence, and discloses a road disease prediction method and device based on an image generative adversarial network, and the method comprises the steps: obtaining a road disease marking image set and a to-be-detected road image set; generating a road disease training image set according to the road disease marking image set; determining a reference road disease judgment result set according to the road disease marking image set and the road disease training image set; determining a road disease loss value set according to the reference road disease judgment result set, the road disease training image set and the road disease marking image set; performing parameter adjustment on the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model; and according to the target image generation type adversarial network model, performing road disease identification on each to-be-detected road image in the to-be-detected road image set to obtain a more accurate target road disease judgment result.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and artificial intelligence technology, and in particular to a road damage prediction method and device based on an image generation adversarial network. Background Art

[0002] In the maintenance and management of road infrastructure, timely and accurate detection of road damage is crucial. With the development of technology, automated road damage detection equipment and methods have emerged. In the process of road damage detection, a large amount of images and data will be generated. However, there is currently a lack of efficient feature extraction and processing methods for the collected road damage images, making it difficult to extract key damage information from them. When analyzing a large amount of data, it is difficult to quickly and accurately identify the development trend and potential risks of the damage, and it is impossible to provide strong support for road maintenance decisions. Therefore, how to improve the accuracy of the road damage identification process has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a road damage prediction method and device based on an image generation adversarial network to solve the technical problem of how to improve the accuracy of a road damage recognition process.

[0004] In a first aspect, a road damage prediction method based on an image generative adversarial network is provided, comprising:

[0005] Obtain a set of road damage marked images with road damage category labels and a set of road images to be detected;

[0006] Based on the generator in the initial image generative adversarial network model, a road disease training image set is generated according to the road disease labeled image set;

[0007] Based on the discriminator in the initial image generative adversarial network model, a reference road disease discrimination result set is determined according to the road disease labeled image set and the road disease training image set;

[0008] Determine a road disease loss value set according to a reference road disease identification result set, a road disease training image set, and a road disease marked image set;

[0009] According to the road damage loss value set, the parameters of the initial image generative adversarial network model are adjusted to obtain the target image generative adversarial network model;

[0010] According to the target image generative adversarial network model, road damage recognition is performed on each road image to be detected in the road image set to obtain the target road damage discrimination result.

[0011] In a second aspect, a road damage prediction device based on an image generative adversarial network is provided, comprising:

[0012] An acquisition module, used to acquire a set of road damage marked images with road damage category labels and a set of road images to be detected;

[0013] A generation module, for generating a road disease training image set based on a generator in an initial image generative adversarial network model and a road disease labeled image set;

[0014] A first determination module is used to determine a reference road disease discrimination result set based on a discriminator in an initial image generative adversarial network model and a road disease marked image set and a road disease training image set;

[0015] A second determination module is used to determine a road disease loss value set based on a reference road disease identification result set, a road disease training image set, and a road disease marked image set;

[0016] The first processing module is used to adjust the parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain the target image generative adversarial network model;

[0017] The second processing module is used to identify road damage on each road image to be detected in the road image set according to the target image generative adversarial network model to obtain a target road damage discrimination result.

[0018] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the road damage prediction method based on an image generative adversarial network are implemented.

[0019] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the road damage prediction method based on the image generative adversarial network are implemented.

[0020] In the scheme implemented by the above-mentioned road disease prediction method and device based on image generative adversarial network, a set of road disease marked images with road disease category labels and a set of road images to be detected are obtained, so as to generate a set of road disease training images based on the generator in the initial image generative adversarial network model according to the set of road disease marked images, and to determine a reference road disease discrimination result set based on the set of road disease marked images and the set of road disease training images based on the discriminator in the initial image generative adversarial network model, so as to further determine a set of road disease loss values ​​based on the reference road disease discrimination result set, the set of road disease training images and the set of road disease marked images, so as to adjust the parameters of the initial image generative adversarial network model according to the set of road disease loss values ​​to obtain a target image generative adversarial network model, and then, according to the target image generative adversarial network model, road disease recognition is performed on each road image to be detected in the set of road images to be detected, so as to obtain a more accurate target road disease discrimination result that better meets the requirements of road disease recognition, which is conducive to achieving the purpose of more accurate and efficient road disease recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 Schematic diagram of an application environment of a road damage prediction method based on an image generative adversarial network in one embodiment of the present invention;

[0023] Figure 2 is a flow chart of a road damage prediction method based on an image generative adversarial network in one embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of a road damage prediction device based on an image generative adversarial network in one embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0026] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The road damage prediction method based on image generative adversarial network provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server through the network. For example, taking the scenario of road damage recognition for road images to be detected as an example, the target user (such as the R&D personnel or management personnel of the image generative adversarial network model) can upload a set of road damage marked images with road damage category labels and a set of road images to be detected through the client.

[0029] Correspondingly, the server can obtain a set of road disease marked images with road disease category labels and a set of road images to be detected through the client, so as to generate a set of road disease training images based on the generator in the initial image generative adversarial network model and the road disease marked image set, and determine a reference road disease discrimination result set based on the road disease marked image set and the road disease training image set based on the discriminator in the initial image generative adversarial network model, so as to further determine a road disease loss value set based on the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, so as to adjust the parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model, and then perform road disease recognition on each road image to be detected in the road image set to be detected according to the target image generative adversarial network model, so as to obtain a more accurate target road disease discrimination result that better meets the requirements of road disease recognition, and feed the target road disease discrimination result back to the client. Correspondingly, the client can receive the target road damage identification result from the server, and can display the target road damage identification result on the client for the target user to query or browse. By adopting the road damage prediction method based on image generative adversarial network provided by this application, the accuracy of road damage identification can be effectively improved, which is conducive to obtaining a more accurate target road damage identification result that better meets the requirements of road damage identification.

[0030] The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 As shown, Figure 2 A schematic flow chart of a road damage prediction method based on an image generative adversarial network provided in an embodiment of the present invention includes the following steps:

[0032] S10: Obtain a set of road damage marked images with road damage category labels and a set of road images to be detected.

[0033] The road damage marked image set may include one or more road damage marked images, and the road damage marked image may be an image marked with a road damage category. In other words, the road damage marked image may be a real road damage marked image with road damage marked. The road damage category label may be a label marked manually, or may be a label marked intelligently by an artificial intelligence system and manually reviewed, and this application does not limit this.

[0034] The set of road images to be detected may include one or more road images to be detected, and the road images to be detected may be images that need to be detected for road damage. In other words, the server can determine whether each road image to be detected in the set of road images to be detected has road damage, and if there is road damage, it is necessary to determine the type of road damage, which is not limited in this application.

[0035] S20: Based on the generator in the initial image generative adversarial network model, a road disease training image set is generated according to the road disease labeled image set.

[0036] The initial image generative adversarial network (GAN) model is a deep learning model architecture consisting of a generator and a discriminator. It is understandable that the initial image generative adversarial network model may be a model without parameter adjustment, and its performance is usually limited.

[0037] The generator can be used to generate realistic road damage training images in an attempt to deceive the discriminator. The generator can be composed of a series of neural network layers, such as a convolutional neural network or a fully connected neural network, to take random noise or potential vectors as input and gradually convert them into road damage training images required by this application, which is not limited by this application. The road damage training image set can include one or more road damage training images, which can be used to indicate images generated based on the generator and according to the road damage labeled image.

[0038] It should be understood that generating a road disease training image set based on a road disease labeled image set based on a generator in an initial image generative adversarial network model refers to a process of generating road disease training images. In step S20, i.e. generating a road disease training image set based on a road disease labeled image set based on a generator in an initial image generative adversarial network model, may include the following steps:

[0039] S21: Based on the generator in the initial image generative adversarial network model, feature extraction is performed on each road disease marked image in the road disease marked image set to obtain a road disease feature vector set;

[0040] S22: performing dimension-upgrading processing on each random noise vector in the random noise vector set according to the road disease feature vector set to obtain a random dimension-upgrading vector set;

[0041] S23: performing feature mapping processing on each random dimension-raising vector in the random dimension-raising vector set according to the road disease feature vector set to obtain a random feature map set;

[0042] S24: performing upsampling processing on each random feature map in the random feature map set according to the road disease feature vector set to obtain a random high-resolution feature map set;

[0043] S25: According to the road damage feature vector set, a road damage feature establishment process is performed on each random high-resolution feature map in the random high-resolution feature map set to obtain a road damage training image set.

[0044] The road damage feature vector set may include one or more road damage feature vectors, which may be used to indicate the road damage feature vector obtained after the feature extraction of the road damage mark image based on the generator. For example, the server may extract key road damage features such as the shape of cracks, the size and depth of potholes, and the texture of damaged road surface from the road damage mark image to convert complex image data into a feature vector representation that is easier to process and analyze.

[0045] Optionally, the server can use methods such as convolutional neural network (CNN) to implement the above feature extraction steps. Among them, CNN can automatically learn different levels of features in the image by combining multiple convolutional layers and pooling layers. For example, for a road disease mark image with road crack disease marks, CNN can extract the edge features, direction, depth and other features of the cracks, and this application does not limit this.

[0046] The random dimension-upgraded vector set may include one or more random dimension-upgraded vectors, which may be used to indicate the vector obtained after the random noise vector is dimensionally upgraded according to the road disease feature vector. The server may implement the above-mentioned dimension-upgraded processing through methods such as a fully connected layer or a convolutional layer, so that the random dimension-upgraded vector obtained after the dimension-upgraded processing may have more information and a higher dimension, so as to facilitate the subsequent feature mapping processing.

[0047] It is understandable that in the process of performing a series of feature transformation operations on the random noise vector, such as the above-mentioned dimensionality increase processing, and the feature mapping processing and convolution processing to be mentioned later, the generator can learn feature patterns (road disease feature vectors) from real road disease marked images, and gradually apply these feature patterns to the shaping process of the random noise vector. Optionally, the random dimensionality increase vector obtained after dimensionality increase can better interact with the road disease feature vector set in subsequent steps, thereby generating more realistic road disease training images.

[0048] The random feature map set may include one or more random feature maps, which may be used to indicate a feature map obtained after feature mapping processing is performed on the random dimension-upgraded vector according to the road damage feature vector. Optionally, the server may perform feature mapping processing using methods such as a convolution layer or a deconvolution layer. For example, the server may fuse the dimension-upgraded random dimension-upgraded vector with the road damage feature vector through a convolution layer to obtain a random feature map having road damage characteristics.

[0049] The random high-resolution feature map set may include one or more random high-resolution feature maps, and the random high-resolution feature map may be used to indicate a feature map obtained after upsampling the random feature map according to the road damage feature vector. Optionally, the server may use a deconvolution layer or interpolation method to perform upsampling. For example, the server may gradually upsample the low-resolution random feature map through the deconvolution layer to obtain a higher-resolution random high-resolution feature map, so that it gradually approaches the resolution of the real road damage marking image.

[0050] The server can further process the high-resolution random feature map according to the road damage feature vector set to establish more accurate road damage features, thereby obtaining a final road damage training image set, so that the generated feature map is more consistent with the actual road damage situation, and the quality and authenticity of the generated image are improved. Optionally, the server can use convolutional layers, activation functions and other methods to perform road damage feature establishment processing. For example, the server can further extract and enhance the high-resolution random feature map through a combination of multiple convolutional layers and activation functions, thereby establishing more accurate road damage features, and then obtaining a road damage training image set.

[0051] Exemplarily, the server can construct the following generator network (taking the road damage mark image with a size of 64x64 pixels and a color image (3 channels) as an example):

[0052] 1. Input layer, used to receive a random noise vector, such as a 100-dimensional random noise vector.

[0053] 2. Hidden layer 1, a fully connected layer, is used for dimensionality increase processing, such as mapping a 100-dimensional random noise vector to 256 dimensions to obtain a random dimensionality increase vector. It is understandable that in order to enable the generator to gradually build a more complex and richer feature representation from low-dimensional random noise in order to generate high-quality road disease images or other types of outputs, the server can help the generator explore and capture more potential features and changes in a higher-dimensional space through dimensionality increase processing, thereby increasing the diversity and realism of the generated image. Optionally, the hidden layer 1 can use a leaky rectified linear unit (Leaky ReLU) for activation processing, which is not limited in this application.

[0054] 3. Hidden layer 2, deconvolution layer, is used for feature mapping processing, such as mapping the 256-dimensional random up-dimensional vector feature to a random feature map of size 4x4x128. It is understandable that the server can further extract higher-level and more abstract feature representations from the original 256-dimensional random up-dimensional vector. Since different dimensions and spatial structures may correspond to different feature attributes, the server can use feature mapping to convert the original features into feature forms that are more suitable for road disease recognition tasks. For example, for the road disease image generation task, the server can extract new features related to the shape, texture, color, etc. of the road disease from the original features. Optionally, the hidden layer 2 can be normalized using a batch normalization function (BatchNormalization) and can be activated using a linear rectifier unit (Rectified Linear Unit, ReLU), which is not limited in this application.

[0055] 4. Hidden layer 3, deconvolution layer, is used for upsampling processing, such as expanding a feature map of size 4x4x128 to a feature map of size 8x8x64. It can be understood that upsampling is an operation that converts a low-resolution feature map into a high-resolution feature map. Here, in the process of expanding a feature map of size 4x4x128 to a feature map of size 8x8x64, the spatial size of the feature map can be increased (i.e., from 4x4 to 8x8), and the number of channels can be adjusted (from 128 to 64). Optionally, the server can use Batch Normalization for normalization and ReLU for activation processing, which is not limited in this application.

[0056] 5. Hidden layer 4, deconvolution layer, is used for upsampling, such as expanding a feature map of size 8x8x64 to a feature map of size 16x16x32. Optionally, the server can use Batch Normalization for normalization and can use ReLU for activation, which is not limited in this application.

[0057] 6. Hidden layer 5, deconvolution layer, is used for upsampling, such as expanding a feature map of size 16x16x32 to a feature map of size 32x32x16. Optionally, the server can use Batch Normalization for normalization and can use ReLU for activation, which is not limited in this application.

[0058] 7. Output layer, deconvolution layer, used for road disease feature establishment processing, such as generating a 64x64x3 feature map. Optionally, the server can use a hyperbolic tangent function (Hyperbolic Tangent, Tanh) for activation processing to map the output value to the range of [-1,1], which is not limited in this application.

[0059] For steps S21-S25, the server extracts features from each road damage marked image in the road damage marked image set based on the generator in the initial image generative adversarial network model to obtain a road damage feature vector set, and performs dimension upscaling on each random noise vector in the random noise vector set according to the road damage feature vector set to obtain a random dimension upscaling vector set, and performs feature mapping on each random dimension upscaling vector in the random dimension upscaling vector set according to the road damage feature vector set to obtain a random feature map set, and further performs upsampling on each random feature map in the random feature map set according to the road damage feature vector set to obtain a random high-resolution feature map set, so that road damage features can be established for each random high-resolution feature map in the random high-resolution feature map set according to the road damage feature vector set to obtain a road disease training image set, which is beneficial for adjusting parameters of the initial image generative adversarial network model based on the road disease training image set in subsequent steps.

[0060] S30: Based on the discriminator in the initial image generative adversarial network model, a reference road disease discrimination result set is determined according to the road disease marked image set and the road disease training image set.

[0061] The discriminator can be used to distinguish whether the input image is a real road damage marking image or a road damage training image generated by the generator. The discriminator usually has a similar structure to the generator, such as the discriminator can be composed of a neural network, but the purpose of the discriminator and the generator is different. The discriminator receives an image (such as a road damage marking image or a road damage training image) as input and outputs a scalar value representing the probability that the input image is a real road damage marking image. The goal of the discriminator can be to judge the authenticity of the image as accurately as possible, thereby providing feedback to the generator to encourage the generator to generate more realistic images.

[0062] The reference road damage discrimination result set may include one or more reference road damage discrimination results, and the reference road damage discrimination result may be used to indicate the discrimination result obtained after the road damage mark image and the road damage training image are discriminated by the discriminator of the initial image generative adversarial network model. It is understandable that the reference road damage discrimination result is not necessarily accurate, so the parameters of the initial image generative adversarial network model may be further adjusted according to the reference road damage discrimination result to obtain a more accurate target image generative adversarial network model that better meets the requirements for road damage discrimination, and the present application does not limit this.

[0063] It should be understood that determining the reference road damage discrimination result set based on the discriminator in the initial image generative adversarial network model and the road damage marked image set and the road damage training image set refers to the process of how the discriminator determines the reference road damage discrimination result. Among them, step S30, that is, determining the reference road damage discrimination result set based on the discriminator in the initial image generative adversarial network model and the road damage marked image set and the road damage training image set, may include the following steps:

[0064] S31: extracting road disease features from each road disease marked image in the road disease marked image set and each road disease training image in the road disease training image set to obtain a road disease feature map set;

[0065] S32: performing a maximum pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage enhanced feature map set;

[0066] S33: performing an average pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage smoothed feature map set;

[0067] S34: each road disease local feature map in the road disease local feature map set is spliced ​​with the corresponding road disease global feature map in the road disease global feature map set to obtain a road disease spliced ​​feature map set;

[0068] S35: performing attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a road disease attention feature map set;

[0069] S36: Perform full connection processing on each road damage attention feature map in the road damage attention feature map set to obtain a reference road damage discrimination result set.

[0070] The road damage feature map set may include one or more road damage feature maps, which may be used to indicate feature maps obtained after road damage feature extraction is performed on a road damage marking image and road damage feature extraction is performed on a road damage training image. The method for extracting road damage features from a road damage training image by the server may adopt methods such as convolutional neural network and principal component analysis, which is not limited in this application.

[0071] The road damage enhancement feature map set may include one or more road damage enhancement feature maps, which can be used to indicate the feature map obtained after the maximum pooling operation is performed on the road damage feature map. The server performs a maximum pooling operation on the road damage feature map, which can highlight the local significant features in the image. For example, if there is an obvious crack edge or the deepest point of a pothole, maximum pooling may capture these areas with higher pixel values; and it helps to strengthen edge information. For example, the pixel value changes at the edge are usually large, and the local maximum value is selected through the maximum pooling operation to more clearly highlight the changes in edge information. Optionally, the maximum pooling operation has a certain anti-interference ability, which can make the extracted features more robust, and has a certain tolerance for lighting changes, slight perspective changes, etc., which helps to reduce the number of parameters and calculations of the model and improve calculation efficiency. This application does not impose any restrictions on this.

[0072] The road damage smooth feature map set may include one or more road damage smooth feature maps, and the road damage smooth feature map may be used to indicate the feature map obtained after the average pooling operation is performed on the road damage feature map. The server performs an average pooling operation on the road damage feature map, which can reduce the sharp changes and noise in the road damage feature map, capture the smooth changes and overall trends of the image, and retain more background information to a certain extent, which helps to improve the stability and generalization ability of the model. For example, in the process of processing the road damage feature map, the average pooling operation can obtain the average intensity, color and other information of the road damage area, and this application does not limit this.

[0073] The road damage splicing feature map set may include one or more road damage splicing feature maps, which can be used to indicate the feature map obtained after splicing the road damage local feature map with the corresponding road damage global feature map. The server can obtain a richer and more comprehensive feature vector by splicing the results of maximum pooling and average pooling, providing input data containing different features for attention calculation, which is conducive to applying the spliced ​​feature vector to different types of attention mechanisms in the subsequent process to adapt to different tasks and data characteristics.

[0074] The road damage attention feature map set may include one or more road damage attention feature maps, and the road damage attention feature map may be used to indicate a feature map obtained after attention calculation is performed on a road damage splicing feature map.

[0075] It should be understood that performing attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain the road damage attention feature map set refers to the process of how to perform attention calculation. Among them, step S35, that is, performing attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain the road damage attention feature map set, may include the following steps:

[0076] S351: performing road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set;

[0077] S352: Performing road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set;

[0078] S353: Perform attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set to obtain a fused attention feature map set;

[0079] S354: performing a deep convolution operation on each fused attention feature map in the fused attention feature map set to obtain a deep attention feature map set;

[0080] S355: Perform a residual connection on each deep attention feature map in the deep attention feature map set to obtain a residual attention feature map set;

[0081] S356: Adaptively activate each residual attention feature map in the residual attention feature map set to obtain a road damage attention feature map set.

[0082] Among them, the local attention feature map set may include one or more local attention feature maps, and the local attention feature map can be used to indicate the feature map obtained after the road disease local attention calculation is performed on the road disease splicing feature map. By introducing the local self-attention mechanism, the server can better focus on the local disease area of ​​the road. For example, for a road image with a wide range, the server can first divide the road image into multiple local areas to perform local self-attention calculations in each local area, and then combine the results of global self-attention in subsequent steps, so that the model can better focus on local disease areas while paying attention to global information.

[0083] The server can first determine the size and shape of the local diseased area of ​​the road according to the specific circumstances. For example, the server can divide the image into non-overlapping rectangular areas, which is not limited in this application. Secondly, for each local diseased area of ​​the road, the server can calculate the corresponding local attention weights respectively. Optionally, the server can adopt a traditional self-attention mechanism, such as the server can adopt a multi-head attention mechanism (Multi-Head Attention, MHA) to calculate the local attention weight. Furthermore, the server can perform a weighted summation of the local attention weight and the value of the corresponding local area to obtain a set of local attention feature maps.

[0084] Optionally, the server performs a road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set, which can be seen in the following formula:

[0085]

[0086] Among them, LocalAttention(T local_t ) can be used to represent the tth local attention feature map in the local attention feature map set, T local_t It can be used to represent the tth road disease splicing feature map, t can be used to represent the index in the road disease splicing feature map set; n×n can be used to represent the number of local disease areas of the road divided by the first road disease splicing feature map; m can be used to represent the index of the number of local disease areas of the road; Attention m () can be used to represent the operation of performing local attention calculation on the local diseased area of ​​the mth road; Q m It can be used to represent the first feature matrix extracted from the local disease area of ​​the mth road, that is, the query matrix of the local disease area of ​​the mth road; K m It can be used to represent the second feature matrix extracted from the local disease area of ​​the mth road, that is, the key matrix of the local disease area of ​​the mth road; V m It can be used to represent the third feature matrix extracted from the local disease area of ​​the m-th road, that is, the value matrix of the local disease area of ​​the m-th road.

[0087] The global attention feature map set may include one or more global attention feature maps, which may be used to indicate a feature map obtained after performing a road disease global attention calculation on a road disease splicing feature map. Optionally, the server may perform global attention calculation in the same or different manner as the local attention calculation, and this application does not limit this.

[0088] Optionally, the server can directly use the self-attention mechanism on the entire input feature map (i.e., the road damage splicing feature map) to calculate the global attention weight of the input feature map. The server performs road damage global attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain the step of obtaining the global attention feature map set, which can be seen in the following formula:

[0089]

[0090] Among them, GlobalAttention(T global_t ) can be used to represent the tth global attention feature map in the global attention feature map set, T global_t It can be used to represent the t-th road disease splicing feature map, and t can be used to represent the index in the road disease splicing feature map set; softmax() can be used to represent the normalization operation; Q can be used to represent the first overall feature matrix extracted from the t-th road disease splicing feature map, that is, the query matrix of the t-th road disease splicing feature map; K can be used to represent the second overall feature matrix extracted from the t-th road disease splicing feature map, that is, the key matrix of the t-th road disease splicing feature map, K T Can be used to represent the transpose of the second overall characteristic matrix; d k It can be used to represent the dimension of the second overall feature matrix; V can be used to represent the third feature matrix extracted from the t-th road disease splicing feature map, that is, the value matrix of the t-th road disease splicing feature map.

[0091] The fused attention feature map set may include one or more fused attention feature maps, which may be used to indicate the feature map obtained after each local attention feature map is fused with the corresponding global attention feature map. Optionally, the server may fuse the local attention feature map with the global attention feature map using a weighted fusion method, which is not limited in this application.

[0092] Optionally, each local attention feature map in the local attention feature map set is fused with the corresponding global attention feature map in the global attention feature map set to obtain a fused attention feature map set, as shown in the following formula:

[0093] FusedAttention(T t )

[0094] =δ×GlobalAttention(T global_t )+ε×LocalAttention(T local_t )

[0095] Among them, FusedAttention(T t ) can be used to represent the tth global attention feature map in the fusion attention feature map set, T t It can be used to represent the tth road damage splicing feature map, t can be used to represent the index in the road damage splicing feature map set; δ can be used to represent the weight coefficient of the global attention feature map; GlobalAttention(T global_t ) can be used to represent the tth global attention feature map in the global attention feature map set, T g1obal_t It can be used to represent the t-th road disease splicing feature map; ε can be used to represent the weight coefficient of the local attention feature map; LocalAttention(T local_t ) can be used to represent the tth local attention feature map in the local attention feature map set, T local_t It can be used to represent the tth road disease splicing feature map.

[0096] The deep attention feature map set may include one or more deep attention feature maps, which can be used to represent the feature map obtained after deep convolution operation on the fused attention feature map. After the server performs self-attention operation on the road disease splicing feature map, it can further pass the feed forward neural network (FFN) to increase the nonlinear expression ability of the model, so as to extract more complex features.

[0097] Optionally, the server can use depthwise separable convolution in FFN to reduce the number of parameters while maintaining good feature extraction capabilities, and introduce residual connections to enhance the information transmission and feature reuse capabilities of FFN, so that the model can better learn the complex changes in road disease characteristics, and further use adaptive activation functions to automatically adjust the activation level according to the input features, thereby improving the nonlinear expression ability of the model.

[0098] Optionally, the server performs a deep convolution operation on each fused attention feature map in the fused attention feature map set to obtain a deep attention feature map set, as shown in the following formula:

[0099]

[0100] Among them, S DW,cIt can be used to represent the deep attention feature map of the cth channel in the deep attention feature map set; c can be used to represent the index of the channel; H can be used to represent the size of the convolution kernel, and each channel uses an H×H convolution kernel to perform convolution operations independently; p can be used to represent the first position of the convolution kernel to be traversed; q can be used to represent the second position of the convolution kernel to be traversed; W DW,c (p,q) can be used to represent the position of the convolution kernel of the cth channel to be traversed; FusedAttention(m+p,n+q,c) can be used to represent the value of the cth channel of the input feature map (i.e., the fused attention feature map) at the position (m+p,n+q); (m,n) can be used to represent each position of the input feature map (i.e., the fused attention feature map) that requires convolution operation.

[0101] The residual attention feature map set may include one or more residual attention feature maps, which can be used to represent the feature map obtained after the deep attention feature map is residually connected. By performing residual connections, the server can make it easier for FFN to learn the residual between input and output, avoid the gradient vanishing problem, and thus accelerate training. This application does not limit this.

[0102] The server can further use adaptive activation functions, such as self-gated activation function (Sigmoid-weighted Linear Unit, SiLU, also known as Swish function), Mish activation function (ASelf Regularized Non-Monotonic Neural Activation Function, Mish), etc. to adaptively activate the residual attention feature map, so that the activation degree can be automatically adjusted according to the input features, thereby obtaining a set of road damage attention feature maps that better meet the needs. This application does not impose any restrictions on this.

[0103] For steps S351-S356, the server performs road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set, performs road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set, and performs attention fusion on each local attention feature map in the local attention feature map set with the corresponding global attention feature map in the global attention feature map set to obtain a fused attention feature map set, and further performs deep convolution operation on each fused attention feature map in the fused attention feature map set to obtain a deep attention feature map set, and performs residual connection on each deep attention feature map in the deep attention feature map set to obtain a residual attention feature map set, so that each residual attention feature map in the residual attention feature map set can be adaptively activated to obtain a road disease attention feature map set, which is conducive to capturing more accurate and comprehensive road disease features.

[0104] Furthermore, after obtaining the road damage attention feature map set, the server can perform full connection processing on each road damage attention feature map to convert the road damage attention feature map into an output form suitable for the road damage identification task, that is, to obtain a reference road damage identification result set. Optionally, the server can perform the above-mentioned full connection processing through a fully connected layer, such as integrating and fusing features through a fully connected hierarchy, and further extracting high-level abstract features, learning complex nonlinear relationships, and outputting the results of specific tasks, such as the probability distribution of different types of damage or the severity level of the damage, etc., and this application does not limit this.

[0105] For example, the server can construct the following discriminator network:

[0106] 1. Input layer, such as receiving road disease training images of size 64x64x3;

[0107] 2. Hidden layer 1, convolutional layer, is used to extract road damage features, such as outputting a road damage feature map with a size of 32x32x64;

[0108] 3. Hidden layer 2, convolution layer, is used to further extract features, such as performing a maximum pooling operation or an average pooling operation, such as outputting a road damage enhanced feature map with a size of 16x16x64 and a road damage smoothed feature map with a size of 16x16x64, and further performing a splicing process, such as outputting a road damage splicing feature map with a size of 16x16x128;

[0109] 4. Hidden layer 3, a hidden layer with an attention mechanism, used for attention calculation, such as outputting a 16x16x128 road disease attention feature map;

[0110] 5. Output layer, fully connected layer, is used to perform fully connected processing and output a value to indicate the probability that the input road damage training image is a real road damage labeled image.

[0111] For steps S31-S36, the server extracts road disease features from each road disease training image in the road disease training image set to obtain a road disease feature map set, performs a maximum pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease enhanced feature map set, and performs an average pooling operation on each road disease feature map in the road disease feature map set to obtain a road disease smoothed feature map set, so that each road disease local feature map in the road disease local feature map set and the corresponding road disease global feature map in the road disease global feature map set can be spliced ​​to obtain a road disease spliced ​​feature map set, and performs attention calculation on each road disease spliced ​​feature map in the road disease spliced ​​feature map set to obtain a road disease attention feature map set, and then each road disease attention feature map in the road disease attention feature map set can be fully connected to obtain a reference road disease discrimination result set, which is beneficial for training a target image generative adversarial network model that meets the requirements more based on the reference road disease discrimination result set in subsequent steps.

[0112] S40: Determine a road damage loss value set according to a reference road damage identification result set, a road damage training image set and a road damage marked image set.

[0113] Among them, the road damage loss value set may include one or more road damage loss values, and the road damage loss value can be used to indicate the loss value calculated based on the constructed loss function after the loss function is constructed according to the reference road damage judgment results, road disease training images and road disease marking images.

[0114] The road damage loss value set can measure the performance of the initial image generative adversarial network model in the road damage recognition task, can reflect the difference between the road damage training image generated by the generator and the real road damage marked image, and can reflect the discriminator's ability to distinguish between the generated image and the real image. Optionally, the road damage loss value can include the loss of the generator and the loss of the discriminator. The loss of the generator can be obtained based on the judgment result of the discriminator on the generated image and the difference between the generated image and the real image; the loss of the discriminator can be obtained based on its ability to correctly classify the real image and the generated image, which is not limited in this application.

[0115] It should be understood that determining the road damage loss value set based on the reference road damage identification result set, the road damage training image set and the road damage marked image set refers to the process of how to calculate the road damage loss value. In step S40, that is, determining the road damage loss value set based on the reference road damage identification result set, the road damage training image set and the road damage marked image set, may include the following steps:

[0116] S41: performing adversarial loss calculation according to each reference road disease discrimination result in the reference road disease discrimination result set to obtain a set of adversarial loss values ​​of the generator;

[0117] S42: performing content loss calculation on each road damage training image in the road damage training image set to obtain a content loss value set of a generator;

[0118] S43: performing content loss calculation according to each road disease training image in the road disease training image set and each road disease marked image in the road disease marked image set to obtain a content loss value set of a generator;

[0119] S44: Calculate the true loss according to each road damage marked image in the road damage marked image set to obtain a true loss value set of the discriminator;

[0120] S45: Calculating false loss according to each road damage training image in the road damage training image set to obtain a false loss value set of a discriminator;

[0121] S46: Determine a discriminant loss value set of the discriminator according to the true loss value set and the false loss value set;

[0122] S47: Determine a road damage loss value set based on the generated loss value set and the discriminated loss value set.

[0123] Among them, the adversarial loss value set may include one or more adversarial loss values, which can be used to indicate the loss value corresponding to the generator obtained after the adversarial loss calculation is performed on the reference road damage discrimination result. In order to make the discriminator judge the image generated by it as a real road damage image, the generator introduces an adversarial loss value to indicate that the generator hopes to maximize the probability that the discriminator judges the image generated by it as real, and this application does not limit this.

[0124] Optionally, the server performs adversarial loss calculation based on each reference road damage discrimination result in the reference road damage discrimination result set to obtain the adversarial loss value set of the generator, which can be seen in the following formula:

[0125] L gen_adv_i= -log(D(G(X i )))

[0126] Among them, L gen_adv_i It can be used to represent the i-th adversarial loss value in the adversarial loss value set of the generator; log() can be used to represent the operation of logarithmic operation; D(G(X i )) can be used to represent the output of the discriminator for the i-th image generated by the generator, that is, the i-th reference road disease discrimination result, that is, the output value of the discriminator; G(X i ) can be used to represent the i-th image generated by the generator, that is, the output value of the generator, which is also the input value of the discriminator; X i It can be used to represent the i-th random noise vector, which is the input value of the generator.

[0127] The content loss value set may include one or more content loss values, which can be used to indicate the corresponding loss value of the generator obtained after the content loss calculation is performed on the road damage training image. The generator can calculate the content loss between the road damage marked image with the road damage category label and the road damage training image generated by the generator to ensure that the generated road damage training image is similar in content to the real road damage marked image, and this application does not limit this. Optionally, the server can calculate the mean squared error (MSE) or the structural similarity index (SSIM) to achieve the calculation of the above-mentioned content loss value, and this application does not limit this.

[0128] Optionally, taking the method of calculating MSE by the server as an example, the server performs content loss calculation based on each road disease training image in the road disease training image set and each road disease marked image in the road disease marked image set to obtain a content loss value set of the generator, which can be seen in the following formula:

[0129]

[0130] Among them, L gen_content_i can be used to represent the i-th content loss value in the content loss value set of the generator, i can be used to represent the loss value index in the content loss value set, and the loss value index in the content loss value set can correspond to the loss value index in the adversarial loss value set; N i can be used to represent the number of pixels in the i-th road disease training image, i can be used to represent the image index in the road disease training image set, and the image index in the road disease training image set can correspond to the loss value index in the content loss value set; j can be used to represent the pixel index in the i-th road disease training image; G(X i) can be used to represent the i-th road disease training image generated by the generator, and i can be used to represent the image index in the road disease training image set; G(X i ) j It can be used to represent the j-th pixel value of the i-th road disease training image generated by the generator; Y i can be used to represent the i-th road damage labeled image corresponding to the i-th road damage training image, (Y i ) j It can be used to represent the j-th pixel value of the i-th road damage marking image corresponding to the i-th road damage training image, and j can be used to represent the pixel index in the i-th road damage marking image, and the pixel index in the i-th road damage marking image corresponds to the pixel index in the i-th road disease training image.

[0131] The generation loss value set may include one or more generation loss values, which may be used to indicate the total loss value corresponding to the generator obtained after the adversarial loss value and the content loss value are merged. The server may perform a weighted sum operation on the adversarial loss value and the content loss value to obtain the total loss of the generator, and then minimize the total loss of the generator and continuously adjust the parameters of the generator so that the generator can generate more realistic road damage images. This application does not impose any restrictions on this.

[0132] Optionally, the server determines the step of generating a loss value set of the generator according to the adversarial loss value set and the content loss value set, as shown in the following formula:

[0133] L gen =L gen_adv +αL gen_content

[0134] Among them, L gen Can be used to represent the set of generated loss values ​​of the generator; L gen_adv It can be used to represent the set of adversarial loss values; α can be used to represent the weight coefficient of the generator loss value, which can be used to balance the contribution of adversarial loss and content loss; L gen_content Can be used to represent a set of content loss values.

[0135] The real loss value set may include one or more real loss values, which can be used to indicate the loss value corresponding to the discriminator obtained after the real loss calculation is performed on the road damage mark image. The goal of the discriminator is to distinguish between the real road damage mark image and the road damage training image generated by the generator. Therefore, the server can calculate the loss value of the discriminator for the real road damage mark image, that is, the above-mentioned real loss value set, and calculate the loss value of the discriminator for the road damage training image generated by the generator, that is, the false loss value set mentioned below, to further improve the recognition ability of the discriminator.

[0136] Optionally, for a real road damage mark image, the output of the discriminator should be as close to 1 as possible to indicate that the image is a real image. The server performs a real loss calculation based on each road damage mark image in the road damage mark image set to obtain a set of real loss values ​​of the discriminator, as shown in the following steps:

[0137] L dis_real_i = -log(D(Y i ))

[0138] Among them, L dis_real_i It can be used to represent the i-th true loss value in the true loss value set of the discriminator; log() can be used to represent the operation of logarithmic operation; D(Y i ) can be used to represent the output of the discriminator for the i-th road damage mark image, that is, the output value of the discriminator; Y i It can be used to represent the i-th road damage mark image, which is the input value of the discriminator.

[0139] The false loss value set may include one or more false loss values, which may be used to indicate the loss value corresponding to the discriminator obtained after the false loss calculation is performed on the road damage training image. Optionally, for the road damage training image generated by the generator, the output of the discriminator should be as close to 0 as possible to indicate that the image is a false image.

[0140] Optionally, the server performs false loss calculation based on each road damage training image in the road damage training image set to obtain a false loss value set of the discriminator, which can be seen in the following steps:

[0141] L dis_fake_i = -log(1-D(G(X i )))

[0142] Among them, L dis_fake_i It can be used to represent the i-th true loss value in the set of false loss values ​​of the discriminator; log() can be used to represent the operation of logarithmic operation; D(G(X i)) can be used to represent the output of the discriminator for the i-th image generated by the generator, that is, the i-th reference road disease discrimination result, that is, the output value of the discriminator; G(X i ) can be used to represent the i-th image generated by the generator, that is, the output value of the generator, which is also the input value of the discriminator; X i It can be used to represent the i-th random noise vector, which is the input value of the generator.

[0143] The discriminant loss value set may include one or more discriminant loss values, which may be used to indicate the total loss value corresponding to the discriminator obtained after the real loss value and the false loss value are fused. The server may perform a weighted sum operation on the real loss value and the false loss value to obtain the total loss of the discriminator, and then minimize the total loss of the discriminator and continuously adjust the parameters of the discriminator so that the discriminator can better distinguish between the real road damage marking image and the generated road damage training image, and this application does not limit this.

[0144] Optionally, the server determines the discriminant loss value set of the discriminator according to the real loss value set and the false loss value set, as shown in the following formula:

[0145] L dis =L dis_real +βL dis_fake

[0146] Among them, L dis Can be used to represent the discriminant loss value set of the discriminator; L dis_real can be used to represent the set of true loss values; β can be used to represent the weight coefficient of the discriminator loss value, which can be used to balance the contribution to the true road damage labeling image and the generated road damage training image; L dis_fake Can be used to represent false loss values.

[0147] The server can further combine the generation loss value set and the discrimination loss value set to determine the road damage loss value set corresponding to the image generative adversarial network model, so as to minimize the loss value of the image generative adversarial network model and adjust the parameters of the generator and the discriminator at the same time, so that the generator can generate more realistic road damage training images, and the discriminator can better distinguish between real road damage marked images and generated road damage training images. This application does not impose any restrictions on this.

[0148] Optionally, the server determines the step of generating a loss value set and determining a loss value set to determine a road damage loss value set, as shown in the following formula:

[0149] L street =L gen +γL dis

[0150] Among them, L street It can be used to represent the road damage loss value set of the image generative adversarial network model; L gen It can be used to represent the set of generated loss values ​​of the generator; γ can be used to represent the weight coefficient of the discriminator loss value, which can be used to balance the contribution of the loss of the generator and the discriminator; L dis Can be used to represent the set of discriminative loss values ​​of the discriminator.

[0151] For steps S41-S47, the server performs adversarial loss calculation according to each reference road disease discrimination result in the reference road disease discrimination result set to obtain the adversarial loss value set of the generator, and performs content loss calculation according to each road disease training image in the road disease training image set to obtain the content loss value set of the generator, and further performs content loss calculation according to each road disease training image in the road disease training image set and each road disease marked image in the road disease marked image set to obtain the content loss value set of the generator, and performs true loss calculation according to each road disease marked image in the road disease marked image set to obtain the true loss value set of the discriminator, and performs false loss calculation according to each road disease training image in the road disease training image set to obtain the false loss value set of the discriminator, so as to further determine the discriminant loss value set of the discriminator according to the true loss value set and the false loss value set, and then determine the road disease loss value set according to the generated loss value set and the discriminant loss value set, which is conducive to determining a more accurate target image generative adversarial network model based on the road disease loss value set in subsequent steps.

[0152] S50: According to the road damage loss value set, the parameters of the initial image generative adversarial network model are adjusted to obtain the target image generative adversarial network model.

[0153] The target image generative adversarial network model may be an image generative adversarial network model obtained by adjusting the parameters of the initial image generative adversarial network model according to the road damage loss value set. It is understandable that the target image generative adversarial network model may be a model with clearer goal orientation, higher accuracy, and greater stability.

[0154] It is understandable that the aforementioned road damage loss value set can provide direction and basis for the parameter adjustment of the model. By analyzing the changes in the road damage loss value, the server can determine which parameters in the model need to be adjusted and how to adjust these parameters to reduce the loss value to improve the performance of the model. For example, if the loss value of the generator is high, it may be necessary to adjust the network structure, model learning rate, regularization parameters, etc. of the generator to improve the quality and realism of the generated image; if the loss value of the discriminator is high, it may be necessary to adjust the network structure of the discriminator, initialize the parameters, etc. to improve the discriminator's discrimination ability, and this application does not limit this.

[0155] After multiple parameter adjustments and iterative training on the server side, when the model loss value is within an acceptable range and the performance on the validation set is good, it can be considered that the target image generative adversarial network model is obtained. In the task of processing road damage images, the generator of this target image generative adversarial network model can generate more realistic road damage training images, and the discriminator can more accurately judge the authenticity of the input image, which greatly improves the performance and reliability of the model.

[0156] S60: According to the target image generative adversarial network model, road damage recognition is performed on each road image to be detected in the road image set to obtain a target road damage discrimination result.

[0157] The target road damage discrimination result may be a discrimination result obtained after road damage recognition is performed on the road image to be detected based on the target image generative adversarial network model. The target road damage discrimination result may include but is not limited to whether the identified road image to be detected has road damage, and if so, the type of road damage, the severity of the road damage, the location of the road damage, etc., which is not limited in this application.

[0158] The target road disease identification results can provide important decision-making basis for road maintenance and management departments. For example, according to the identified disease type and severity, the management department can formulate a corresponding maintenance plan and arrange maintenance personnel to carry out timely repairs. The target road disease identification results can be used for data analysis and prediction. For example, by statistically analyzing the disease identification results of a large number of road images, the distribution and development trend of road diseases can be understood. The target road disease identification results can be combined with the intelligent transportation system to provide vehicles with real-time road condition information. For example, the vehicle navigation system can provide the driver with the best driving route based on the identified road disease conditions to avoid passing through sections with serious diseases. This application does not limit this.

[0159] It can be seen that in the above scheme, the server can obtain a set of road disease marked images with road disease category labels and a set of road images to be detected, and can generate a set of road disease training images based on the generator in the initial image generative adversarial network model according to the set of road disease marked images, and determine a reference road disease discrimination result set based on the set of road disease marked images and the set of road disease training images based on the discriminator in the initial image generative adversarial network model, so as to determine a set of road disease loss values ​​according to the reference road disease discrimination result set, the set of road disease training images and the set of road disease marked images, and further adjust the parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model, and then perform road disease recognition on each road image to be detected in the set of road images to be detected according to the target image generative adversarial network model, so as to obtain a target road disease discrimination result that is more accurate and more in line with the road disease recognition requirements.

[0160] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0161] In one embodiment, a road damage prediction device based on an image generation adversarial network is provided, and the road damage prediction device based on an image generation adversarial network corresponds one-to-one to the road damage prediction method based on an image generation adversarial network in the above embodiment. Figure 3 As shown, the road damage prediction device based on image generative adversarial network includes an acquisition module 101, a generation module 102, a first determination module 103, a second determination module 104, a first processing module 105 and a second processing module 106. The functional modules are described in detail as follows:

[0162] An acquisition module 101 is used to acquire a set of road damage marked images with road damage category labels and a set of road images to be detected;

[0163] A generating module 102 is used to generate a road disease training image set based on a generator in an initial image generative adversarial network model and a road disease labeled image set;

[0164] The first determination module 103 is used to determine a reference road disease discrimination result set based on the discriminator in the initial image generative adversarial network model and the road disease marked image set and the road disease training image set;

[0165] A second determination module 104 is used to determine a road damage loss value set according to a reference road damage identification result set, a road damage training image set and a road damage marked image set;

[0166] The first processing module 105 is used to adjust the parameters of the initial image generative adversarial network model according to the road disease loss value set to obtain a target image generative adversarial network model;

[0167] The second processing module 106 is used to perform road damage recognition on each road image to be detected in the road image set according to the target image generative adversarial network model to obtain a target road damage discrimination result.

[0168] In one embodiment, the generating module 102 is used to generate a road damage training image set based on the generator in the initial image generative adversarial network model and the road damage labeled image set, specifically for:

[0169] Based on the generator in the initial image generative adversarial network model, feature extraction is performed on each road disease marked image in the road disease marked image set to obtain a road disease feature vector set;

[0170] According to the road disease feature vector set, each random noise vector in the random noise vector set is subjected to dimension-raising processing to obtain a random dimension-raising vector set;

[0171] According to the road disease feature vector set, feature mapping is performed on each random dimension-raising vector in the random dimension-raising vector set to obtain a random feature map set;

[0172] According to the road disease feature vector set, up-sampling is performed on each random feature map in the random feature map set to obtain a random high-resolution feature map set;

[0173] According to the road damage feature vector set, each random high-resolution feature map in the random high-resolution feature map set is processed to establish the road damage feature, so as to obtain a road damage training image set.

[0174] In one embodiment, the first determination module 103 is used to determine a reference road damage discrimination result set based on the discriminator in the initial image generative adversarial network model and the road damage marked image set and the road damage training image set, specifically for:

[0175] Extract road damage features from each road damage marked image in the road damage marked image set and each road damage training image in the road damage training image set to obtain a road damage feature map set;

[0176] Performing a maximum pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage enhanced feature map set;

[0177] Performing an average pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage smooth feature map set;

[0178] Each road disease local feature map in the road disease local feature map set is spliced ​​with the corresponding road disease global feature map in the road disease global feature map set to obtain a road disease spliced ​​feature map set;

[0179] Performing attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a road disease attention feature map set;

[0180] Each road damage attention feature map in the road damage attention feature map set is fully connected to obtain a reference road damage discrimination result set.

[0181] In one embodiment, the first determination module 103 is used to perform attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain a road damage attention feature map set, specifically for:

[0182] Performing road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set;

[0183] Performing road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set;

[0184] Perform attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set to obtain a fused attention feature map set;

[0185] Perform a deep convolution operation on each fused attention feature map in the fused attention feature map set to obtain a deep attention feature map set;

[0186] Perform a residual connection on each deep attention feature map in the deep attention feature map set to obtain a residual attention feature map set;

[0187] Each residual attention feature map in the residual attention feature map set is adaptively activated to obtain a road damage attention feature map set.

[0188] In one embodiment, the second determination module 104 is used to determine a road damage loss value set based on the reference road damage identification result set, the road damage training image set and the road damage marked image set, specifically for:

[0189] Calculate the adversarial loss based on each reference road disease discrimination result in the reference road disease discrimination result set to obtain a set of adversarial loss values ​​of the generator;

[0190] Calculate the content loss based on each road disease training image in the road disease training image set and each road disease labeled image in the road disease labeled image set to obtain a content loss value set of the generator;

[0191] Determine a generation loss value set of the generator according to the adversarial loss value set and the content loss value set;

[0192] Calculate the true loss for each road damage marked image in the road damage marked image set to obtain a true loss value set of the discriminator;

[0193] Calculate the false loss according to each road disease training image in the road disease training image set to obtain a false loss value set of the discriminator;

[0194] Determine a discriminant loss value set of the discriminator according to the real loss value set and the false loss value set;

[0195] According to the generated loss value set and the discriminated loss value set, a road damage loss value set is determined.

[0196] The present invention provides a road disease prediction device based on an image generative adversarial network. By acquiring a road disease marked image set with road disease category labels and a road image set to be detected, a road disease training image set can be generated based on a generator in an initial image generative adversarial network model according to the road disease marked image set, and a reference road disease discrimination result set can be determined based on a discriminator in the initial image generative adversarial network model according to the road disease marked image set and the road disease training image set. Thus, a road disease loss value set can be determined according to the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, and further according to the road disease loss value set, the parameters of the initial image generative adversarial network model can be adjusted to obtain a target image generative adversarial network model, and then, according to the target image generative adversarial network model, road disease recognition is performed on each road image to be detected in the road image set to be detected, so as to obtain a more accurate target road disease discrimination result that better meets the road disease recognition requirements.

[0197] For the specific definition of the road damage prediction device based on the image generative adversarial network, please refer to the definition of the road damage prediction method based on the image generative adversarial network above, which will not be repeated here. Each module in the above-mentioned road damage prediction device based on the image generative adversarial network can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0198] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the service side of a road disease prediction method based on an image generative adversarial network.

[0199] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a road disease prediction method based on an image generative adversarial network.

[0200] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0201] Obtain a set of road damage marked images with road damage category labels and a set of road images to be detected;

[0202] Based on the generator in the initial image generative adversarial network model, a road disease training image set is generated according to the road disease labeled image set;

[0203] Based on the discriminator in the initial image generative adversarial network model, a reference road disease discrimination result set is determined according to the road disease labeled image set and the road disease training image set;

[0204] Determine a road disease loss value set according to a reference road disease identification result set, a road disease training image set, and a road disease marked image set;

[0205] According to the road damage loss value set, the parameters of the initial image generative adversarial network model are adjusted to obtain the target image generative adversarial network model;

[0206] According to the target image generative adversarial network model, road damage recognition is performed on each road image to be detected in the road image set to obtain the target road damage discrimination result.

[0207] The present invention provides a computer device, which can generate a road disease training image set based on the road disease marked image set and the road disease training image set based on the generator in the initial image generative adversarial network model by acquiring a road disease marked image set with road disease category labels and a road image set to be detected, and determine a reference road disease discrimination result set based on the road disease marked image set and the road disease training image set based on the discriminator in the initial image generative adversarial network model, so as to determine a road disease loss value set based on the reference road disease discrimination result set, the road disease training image set and the road disease marked image set, and further adjust the parameters of the initial image generative adversarial network model based on the road disease loss value set to obtain a target image generative adversarial network model, and then perform road disease recognition on each road image to be detected in the road image set to be detected based on the target image generative adversarial network model, so as to obtain a target road disease discrimination result that is more accurate and more in line with the road disease recognition requirements.

[0208] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0209] Obtain a set of road damage marked images with road damage category labels and a set of road images to be detected;

[0210] Based on the generator in the initial image generative adversarial network model, a road disease training image set is generated according to the road disease labeled image set;

[0211] Based on the discriminator in the initial image generative adversarial network model, a reference road disease discrimination result set is determined according to the road disease labeled image set and the road disease training image set;

[0212] Determine a road disease loss value set according to a reference road disease identification result set, a road disease training image set, and a road disease marked image set;

[0213] According to the road damage loss value set, the parameters of the initial image generative adversarial network model are adjusted to obtain the target image generative adversarial network model;

[0214] According to the target image generative adversarial network model, road damage recognition is performed on each road image to be detected in the road image set to obtain the target road damage discrimination result.

[0215] The present invention provides a computer-readable storage medium. By acquiring a set of road disease marked images with road disease category labels and a set of road images to be detected, a set of road disease training images can be generated based on a generator in an initial image generative adversarial network model according to the set of road disease marked images, and a set of reference road disease discrimination results can be determined based on a discriminator in the initial image generative adversarial network model according to the set of road disease marked images and the set of road disease training images. Thus, a set of road disease loss values ​​can be determined based on the reference road disease discrimination result set, the set of road disease training images and the set of road disease marked images, and further parameters of the initial image generative adversarial network model can be adjusted according to the road disease loss value set to obtain a target image generative adversarial network model, and then road disease recognition is performed on each road image to be detected in the set of road images to be detected according to the target image generative adversarial network model, so as to obtain a target road disease discrimination result that is more accurate and more in line with the road disease recognition requirements.

[0216] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0217] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0218] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0219] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A road damage prediction method based on image generative adversarial network, characterized in that: The method comprises: Obtain a set of road damage marked images with road damage category labels and a set of road images to be detected; Based on the generator in the initial image generative adversarial network model, a road disease training image set is generated according to the road disease labeled image set; Based on the discriminator in the initial image generative adversarial network model, and according to the road disease marked image set and the road disease training image set, a reference road disease discrimination result set is determined; Determining a road damage loss value set according to the reference road damage identification result set, the road damage training image set and the road damage marked image set; According to the road damage loss value set, adjusting parameters of the initial image generative adversarial network model to obtain a target image generative adversarial network model; According to the target image generative adversarial network model, road damage identification is performed on each road image to be detected in the set of road images to obtain a target road damage discrimination result.

2. The road damage prediction method based on image generative adversarial network according to claim 1 is characterized in that: The generator in the initial image-based generative adversarial network model generates a road disease training image set according to the road disease marked image set, including: Based on the generator in the initial image generative adversarial network model, feature extraction is performed on each road damage marked image in the road damage marked image set to obtain a road damage feature vector set; According to the road disease feature vector set, each random noise vector in the random noise vector set is subjected to dimension-raising processing to obtain a random dimension-raising vector set; According to the road damage feature vector set, feature mapping is performed on each random dimension-raising vector in the random dimension-raising vector set to obtain a random feature map set; According to the road damage feature vector set, upsampling each random feature map in the random feature map set is performed to obtain a random high-resolution feature map set; According to the road damage feature vector set, each random high-resolution feature map in the random high-resolution feature map set is processed to establish a road damage feature, so as to obtain a road damage training image set.

3. The road damage prediction method based on image generative adversarial network according to claim 2 is characterized in that: The discriminator in the generative adversarial network model based on the initial image determines a reference road damage discrimination result set according to the road damage marked image set and the road damage training image set, including: Extract road damage features from each road damage marked image in the road damage marked image set and each road damage training image in the road damage training image set to obtain a road damage feature map set; Performing a maximum pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage enhanced feature map set; Performing an average pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage smooth feature map set; Each road damage local feature map in the road damage local feature map set is spliced ​​with the corresponding road damage global feature map in the road damage global feature map set to obtain a road damage spliced ​​feature map set; Performing attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain a road damage attention feature map set; Each road damage attention feature map in the road damage attention feature map set is fully connected to obtain a reference road damage discrimination result set.

4. The road damage prediction method based on image generative adversarial network according to claim 3 is characterized in that: The step of performing attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain a road damage attention feature map set includes: Performing road disease local attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a local attention feature map set; Performing road disease global attention calculation on each road disease splicing feature map in the road disease splicing feature map set to obtain a global attention feature map set; Performing attention fusion on each local attention feature map in the local attention feature map set and the corresponding global attention feature map in the global attention feature map set to obtain a fused attention feature map set; Performing a deep convolution operation on each fused attention feature map in the fused attention feature map set to obtain a deep attention feature map set; Performing a residual connection on each deep attention feature map in the deep attention feature map set to obtain a residual attention feature map set; Each residual attention feature map in the residual attention feature map set is adaptively activated to obtain a road damage attention feature map set.

5. The road damage prediction method based on image generative adversarial network according to claim 4 is characterized in that: The step of determining a road damage loss value set according to the reference road damage identification result set, the road damage training image set and the road damage marked image set comprises: Performing adversarial loss calculation according to each reference road damage discrimination result in the reference road damage discrimination result set to obtain an adversarial loss value set of the generator; Performing content loss calculation based on each road damage training image in the road damage training image set and each road damage marked image in the road damage marked image set to obtain a content loss value set of the generator; Determining a generation loss value set of the generator according to the adversarial loss value set and the content loss value set; Calculating the true loss of each road damage mark image in the road damage mark image set to obtain a true loss value set of the discriminator; Calculating false loss based on each road damage training image in the road damage training image set to obtain a false loss value set of the discriminator; Determine a discriminant loss value set of the discriminator according to the true loss value set and the false loss value set; A road damage loss value set is determined based on the generated loss value set and the discriminated loss value set.

6. A road damage prediction device based on an image generative adversarial network, characterized in that: The road damage prediction device based on image generative adversarial network includes: An acquisition module, used to acquire a set of road damage marked images with road damage category labels and a set of road images to be detected; A generation module, configured to generate a road disease training image set based on the generator in the initial image generative adversarial network model and the road disease labeled image set; A first determination module is used to determine a reference road disease discrimination result set based on the discriminator in the initial image generative adversarial network model and the road disease marked image set and the road disease training image set; A second determination module is used to determine a road damage loss value set according to the reference road damage identification result set, the road damage training image set and the road damage marked image set; A first processing module is used to adjust the parameters of the initial image generative adversarial network model according to the road damage loss value set to obtain a target image generative adversarial network model; The second processing module is used to perform road damage recognition on each road image to be detected in the set of road images to be detected according to the target image generative adversarial network model to obtain a target road damage discrimination result.

7. The road damage prediction device based on image generative adversarial network according to claim 6 is characterized in that: The generation module is used to generate a road disease training image set based on the generator in the initial image generative adversarial network model and the road disease marked image set, and is specifically used to: Based on the generator in the initial image generative adversarial network model, feature extraction is performed on each road damage marked image in the road damage marked image set to obtain a road damage feature vector set; According to the road disease feature vector set, each random noise vector in the random noise vector set is subjected to dimension-raising processing to obtain a random dimension-raising vector set; According to the road damage feature vector set, feature mapping is performed on each random dimension-raising vector in the random dimension-raising vector set to obtain a random feature map set; According to the road damage feature vector set, upsampling each random feature map in the random feature map set is performed to obtain a random high-resolution feature map set; According to the road damage feature vector set, each random high-resolution feature map in the random high-resolution feature map set is processed to establish a road damage feature, so as to obtain a road damage training image set.

8. The road damage prediction device based on image generative adversarial network according to claim 7 is characterized in that: The first determination module is used to determine a reference road damage discrimination result set based on the discriminator in the initial image generative adversarial network model and the road damage marked image set and the road damage training image set, and is specifically used to: Extract road damage features from each road damage marked image in the road damage marked image set and each road damage training image in the road damage training image set to obtain a road damage feature map set; Performing a maximum pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage enhanced feature map set; Performing an average pooling operation on each road damage feature map in the road damage feature map set to obtain a road damage smooth feature map set; Each road damage local feature map in the road damage local feature map set is spliced ​​with the corresponding road damage global feature map in the road damage global feature map set to obtain a road damage spliced ​​feature map set; Performing attention calculation on each road damage splicing feature map in the road damage splicing feature map set to obtain a road damage attention feature map set; Each road damage attention feature map in the road damage attention feature map set is fully connected to obtain a reference road damage discrimination result set.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the road damage prediction method based on the image generative adversarial network as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the road damage prediction method based on an image generative adversarial network as described in any one of claims 1 to 5 is implemented.

Images